jung-geun/PSO
1
0
Fork 0
mirror of https://github.com/jung-geun/PSO.git synced 2025-12-19 20:44:39 +09:00
Code Issues Packages Projects Releases Wiki Activity

23-10-25

Browse Source 
version 1.0.3
최고 점수 클래스 변수로 변경
log 저장 위치 고정
bean, seeds 데이터셋 추가 실험
This commit is contained in:
jung-geun
2023-10-25 12:53:23 +09:00
parent c8741dcd6d
commit 389027409d
13 changed files with 19828 additions and 225 deletions
Download Patch File Download Diff File Expand all files Collapse all files

69
bean.py Normal file
View File

@@ -0,0 +1,69 @@
import os
import pandas as pd
import tensorflow as tf
from sklearn.model_selection import train_test_split
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.layers import Dense
from tensorflow.keras.models import Sequential
from tensorflow.keras.utils import to_categorical
from ucimlrepo import fetch_ucirepo
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true'
def make_model():
model = Sequential()
model.add(Dense(12, input_dim=16, activation='relu'))
model.add(Dense(8, activation='relu'))
model.add(Dense(7, activation='softmax'))
return model
def get_data():
# fetch dataset
dry_bean_dataset = fetch_ucirepo(id=602)
# data (as pandas dataframes)
X = dry_bean_dataset.data.features
y = dry_bean_dataset.data.targets
x = X.to_numpy()
# object to categorical
x = x.astype('float32')
y_class = to_categorical(y)
# metadata
# print(dry_bean_dataset.metadata)
# variable information
# print(dry_bean_dataset.variables)
# print(X.head())
# print(y.head())
# y_class = to_categorical(y)
x_train, x_test, y_train, y_test = train_test_split(
x, y_class, test_size=0.2, random_state=42, shuffle=True)
return x_train, x_test, y_train, y_test
x_train, x_test, y_train, y_test = get_data()
model = make_model()
early_stopping = keras.callbacks.EarlyStopping(
patience=10, min_delta=0.001, restore_best_weights=True)
model.compile(loss='sparse_categorical_crossentropy',
optimizer='adam', metrics=['accuracy', "mse"])
model.summary()
history = model.fit(x_train, y_train, epochs=150,
batch_size=10, callbacks=[early_stopping])
score = model.evaluate(x_test, y_test, verbose=2)

19020
data/magic+gamma+telescope/magic04.data Executable file
View File

File diff suppressed because it is too large Load Diff

111
data/magic+gamma+telescope/magic04.names Executable file
View File

@@ -0,0 +1,111 @@
1. Title of Database: MAGIC gamma telescope data 2004
2. Sources:
(a) Original owner of the database:
R. K. Bock
Major Atmospheric Gamma Imaging Cherenkov Telescope project (MAGIC)
http://wwwmagic.mppmu.mpg.de
rkb@mail.cern.ch
(b) Donor:
P. Savicky
Institute of Computer Science, AS of CR
Czech Republic
savicky@cs.cas.cz
(c) Date received: May 2007
3. Past Usage:
(a) Bock, R.K., Chilingarian, A., Gaug, M., Hakl, F., Hengstebeck, T.,
Jirina, M., Klaschka, J., Kotrc, E., Savicky, P., Towers, S.,
Vaicilius, A., Wittek W. (2004).
Methods for multidimensional event classification: a case study
using images from a Cherenkov gamma-ray telescope.
Nucl.Instr.Meth. A, 516, pp. 511-528.
(b) P. Savicky, E. Kotrc.
Experimental Study of Leaf Confidences for Random Forest.
Proceedings of COMPSTAT 2004, In: Computational Statistics.
(Ed.: Antoch J.) - Heidelberg, Physica Verlag 2004, pp. 1767-1774.
(c) J. Dvorak, P. Savicky.
Softening Splits in Decision Trees Using Simulated Annealing.
Proceedings of ICANNGA 2007, Warsaw, (Ed.: Beliczynski et. al),
Part I, LNCS 4431, pp. 721-729.
4. Relevant Information:
The data are MC generated (see below) to simulate registration of high energy
gamma particles in a ground-based atmospheric Cherenkov gamma telescope using the
imaging technique. Cherenkov gamma telescope observes high energy gamma rays,
taking advantage of the radiation emitted by charged particles produced
inside the electromagnetic showers initiated by the gammas, and developing in the
atmosphere. This Cherenkov radiation (of visible to UV wavelengths) leaks
through the atmosphere and gets recorded in the detector, allowing reconstruction
of the shower parameters. The available information consists of pulses left by
the incoming Cherenkov photons on the photomultiplier tubes, arranged in a
plane, the camera. Depending on the energy of the primary gamma, a total of
few hundreds to some 10000 Cherenkov photons get collected, in patterns
(called the shower image), allowing to discriminate statistically those
caused by primary gammas (signal) from the images of hadronic showers
initiated by cosmic rays in the upper atmosphere (background).
Typically, the image of a shower after some pre-processing is an elongated
cluster. Its long axis is oriented towards the camera center if the shower axis
is parallel to the telescope's optical axis, i.e. if the telescope axis is
directed towards a point source. A principal component analysis is performed
in the camera plane, which results in a correlation axis and defines an ellipse.
If the depositions were distributed as a bivariate Gaussian, this would be
an equidensity ellipse. The characteristic parameters of this ellipse
(often called Hillas parameters) are among the image parameters that can be
used for discrimination. The energy depositions are typically asymmetric
along the major axis, and this asymmetry can also be used in discrimination.
There are, in addition, further discriminating characteristics, like the
extent of the cluster in the image plane, or the total sum of depositions.
The data set was generated by a Monte Carlo program, Corsika, described in
D. Heck et al., CORSIKA, A Monte Carlo code to simulate extensive air showers,
Forschungszentrum Karlsruhe FZKA 6019 (1998).
The program was run with parameters allowing to observe events with energies down
to below 50 GeV.
5. Number of Instances: 19020
6. Number of Attributes: 11 (including the class)
7. Attribute information:
1. fLength: continuous # major axis of ellipse [mm]
2. fWidth: continuous # minor axis of ellipse [mm]
3. fSize: continuous # 10-log of sum of content of all pixels [in #phot]
4. fConc: continuous # ratio of sum of two highest pixels over fSize [ratio]
5. fConc1: continuous # ratio of highest pixel over fSize [ratio]
6. fAsym: continuous # distance from highest pixel to center, projected onto major axis [mm]
7. fM3Long: continuous # 3rd root of third moment along major axis [mm]
8. fM3Trans: continuous # 3rd root of third moment along minor axis [mm]
9. fAlpha: continuous # angle of major axis with vector to origin [deg]
10. fDist: continuous # distance from origin to center of ellipse [mm]
11. class: g,h # gamma (signal), hadron (background)
8. Missing Attribute Values: None
9. Class Distribution:
g = gamma (signal): 12332
h = hadron (background): 6688
For technical reasons, the number of h events is underestimated.
In the real data, the h class represents the majority of the events.
The simple classification accuracy is not meaningful for this data, since
classifying a background event as signal is worse than classifying a signal
event as background. For comparison of different classifiers an ROC curve
has to be used. The relevant points on this curve are those, where the
probability of accepting a background event as signal is below one of the
following thresholds: 0.01, 0.02, 0.05, 0.1, 0.2 depending on the required
quality of the sample of the accepted events for different experiments.

210
data/seeds/seeds_dataset.txt Executable file
View File

@@ -0,0 +1,210 @@
15.26 14.84 0.871 5.763 3.312 2.221 5.22 1
14.88 14.57 0.8811 5.554 3.333 1.018 4.956 1
14.29 14.09 0.905 5.291 3.337 2.699 4.825 1
13.84 13.94 0.8955 5.324 3.379 2.259 4.805 1
16.14 14.99 0.9034 5.658 3.562 1.355 5.175 1
14.38 14.21 0.8951 5.386 3.312 2.462 4.956 1
14.69 14.49 0.8799 5.563 3.259 3.586 5.219 1
14.11 14.1 0.8911 5.42 3.302 2.7 5 1
16.63 15.46 0.8747 6.053 3.465 2.04 5.877 1
16.44 15.25 0.888 5.884 3.505 1.969 5.533 1
15.26 14.85 0.8696 5.714 3.242 4.543 5.314 1
14.03 14.16 0.8796 5.438 3.201 1.717 5.001 1
13.89 14.02 0.888 5.439 3.199 3.986 4.738 1
13.78 14.06 0.8759 5.479 3.156 3.136 4.872 1
13.74 14.05 0.8744 5.482 3.114 2.932 4.825 1
14.59 14.28 0.8993 5.351 3.333 4.185 4.781 1
13.99 13.83 0.9183 5.119 3.383 5.234 4.781 1
15.69 14.75 0.9058 5.527 3.514 1.599 5.046 1
14.7 14.21 0.9153 5.205 3.466 1.767 4.649 1
12.72 13.57 0.8686 5.226 3.049 4.102 4.914 1
14.16 14.4 0.8584 5.658 3.129 3.072 5.176 1
14.11 14.26 0.8722 5.52 3.168 2.688 5.219 1
15.88 14.9 0.8988 5.618 3.507 0.7651 5.091 1
12.08 13.23 0.8664 5.099 2.936 1.415 4.961 1
15.01 14.76 0.8657 5.789 3.245 1.791 5.001 1
16.19 15.16 0.8849 5.833 3.421 0.903 5.307 1
13.02 13.76 0.8641 5.395 3.026 3.373 4.825 1
12.74 13.67 0.8564 5.395 2.956 2.504 4.869 1
14.11 14.18 0.882 5.541 3.221 2.754 5.038 1
13.45 14.02 0.8604 5.516 3.065 3.531 5.097 1
13.16 13.82 0.8662 5.454 2.975 0.8551 5.056 1
15.49 14.94 0.8724 5.757 3.371 3.412 5.228 1
14.09 14.41 0.8529 5.717 3.186 3.92 5.299 1
13.94 14.17 0.8728 5.585 3.15 2.124 5.012 1
15.05 14.68 0.8779 5.712 3.328 2.129 5.36 1
16.12 15 0.9 5.709 3.485 2.27 5.443 1
16.2 15.27 0.8734 5.826 3.464 2.823 5.527 1
17.08 15.38 0.9079 5.832 3.683 2.956 5.484 1
14.8 14.52 0.8823 5.656 3.288 3.112 5.309 1
14.28 14.17 0.8944 5.397 3.298 6.685 5.001 1
13.54 13.85 0.8871 5.348 3.156 2.587 5.178 1
13.5 13.85 0.8852 5.351 3.158 2.249 5.176 1
13.16 13.55 0.9009 5.138 3.201 2.461 4.783 1
15.5 14.86 0.882 5.877 3.396 4.711 5.528 1
15.11 14.54 0.8986 5.579 3.462 3.128 5.18 1
13.8 14.04 0.8794 5.376 3.155 1.56 4.961 1
15.36 14.76 0.8861 5.701 3.393 1.367 5.132 1
14.99 14.56 0.8883 5.57 3.377 2.958 5.175 1
14.79 14.52 0.8819 5.545 3.291 2.704 5.111 1
14.86 14.67 0.8676 5.678 3.258 2.129 5.351 1
14.43 14.4 0.8751 5.585 3.272 3.975 5.144 1
15.78 14.91 0.8923 5.674 3.434 5.593 5.136 1
14.49 14.61 0.8538 5.715 3.113 4.116 5.396 1
14.33 14.28 0.8831 5.504 3.199 3.328 5.224 1
14.52 14.6 0.8557 5.741 3.113 1.481 5.487 1
15.03 14.77 0.8658 5.702 3.212 1.933 5.439 1
14.46 14.35 0.8818 5.388 3.377 2.802 5.044 1
14.92 14.43 0.9006 5.384 3.412 1.142 5.088 1
15.38 14.77 0.8857 5.662 3.419 1.999 5.222 1
12.11 13.47 0.8392 5.159 3.032 1.502 4.519 1
11.42 12.86 0.8683 5.008 2.85 2.7 4.607 1
11.23 12.63 0.884 4.902 2.879 2.269 4.703 1
12.36 13.19 0.8923 5.076 3.042 3.22 4.605 1
13.22 13.84 0.868 5.395 3.07 4.157 5.088 1
12.78 13.57 0.8716 5.262 3.026 1.176 4.782 1
12.88 13.5 0.8879 5.139 3.119 2.352 4.607 1
14.34 14.37 0.8726 5.63 3.19 1.313 5.15 1
14.01 14.29 0.8625 5.609 3.158 2.217 5.132 1
14.37 14.39 0.8726 5.569 3.153 1.464 5.3 1
12.73 13.75 0.8458 5.412 2.882 3.533 5.067 1
17.63 15.98 0.8673 6.191 3.561 4.076 6.06 2
16.84 15.67 0.8623 5.998 3.484 4.675 5.877 2
17.26 15.73 0.8763 5.978 3.594 4.539 5.791 2
19.11 16.26 0.9081 6.154 3.93 2.936 6.079 2
16.82 15.51 0.8786 6.017 3.486 4.004 5.841 2
16.77 15.62 0.8638 5.927 3.438 4.92 5.795 2
17.32 15.91 0.8599 6.064 3.403 3.824 5.922 2
20.71 17.23 0.8763 6.579 3.814 4.451 6.451 2
18.94 16.49 0.875 6.445 3.639 5.064 6.362 2
17.12 15.55 0.8892 5.85 3.566 2.858 5.746 2
16.53 15.34 0.8823 5.875 3.467 5.532 5.88 2
18.72 16.19 0.8977 6.006 3.857 5.324 5.879 2
20.2 16.89 0.8894 6.285 3.864 5.173 6.187 2
19.57 16.74 0.8779 6.384 3.772 1.472 6.273 2
19.51 16.71 0.878 6.366 3.801 2.962 6.185 2
18.27 16.09 0.887 6.173 3.651 2.443 6.197 2
18.88 16.26 0.8969 6.084 3.764 1.649 6.109 2
18.98 16.66 0.859 6.549 3.67 3.691 6.498 2
21.18 17.21 0.8989 6.573 4.033 5.78 6.231 2
20.88 17.05 0.9031 6.45 4.032 5.016 6.321 2
20.1 16.99 0.8746 6.581 3.785 1.955 6.449 2
18.76 16.2 0.8984 6.172 3.796 3.12 6.053 2
18.81 16.29 0.8906 6.272 3.693 3.237 6.053 2
18.59 16.05 0.9066 6.037 3.86 6.001 5.877 2
18.36 16.52 0.8452 6.666 3.485 4.933 6.448 2
16.87 15.65 0.8648 6.139 3.463 3.696 5.967 2
19.31 16.59 0.8815 6.341 3.81 3.477 6.238 2
18.98 16.57 0.8687 6.449 3.552 2.144 6.453 2
18.17 16.26 0.8637 6.271 3.512 2.853 6.273 2
18.72 16.34 0.881 6.219 3.684 2.188 6.097 2
16.41 15.25 0.8866 5.718 3.525 4.217 5.618 2
17.99 15.86 0.8992 5.89 3.694 2.068 5.837 2
19.46 16.5 0.8985 6.113 3.892 4.308 6.009 2
19.18 16.63 0.8717 6.369 3.681 3.357 6.229 2
18.95 16.42 0.8829 6.248 3.755 3.368 6.148 2
18.83 16.29 0.8917 6.037 3.786 2.553 5.879 2
18.85 16.17 0.9056 6.152 3.806 2.843 6.2 2
17.63 15.86 0.88 6.033 3.573 3.747 5.929 2
19.94 16.92 0.8752 6.675 3.763 3.252 6.55 2
18.55 16.22 0.8865 6.153 3.674 1.738 5.894 2
18.45 16.12 0.8921 6.107 3.769 2.235 5.794 2
19.38 16.72 0.8716 6.303 3.791 3.678 5.965 2
19.13 16.31 0.9035 6.183 3.902 2.109 5.924 2
19.14 16.61 0.8722 6.259 3.737 6.682 6.053 2
20.97 17.25 0.8859 6.563 3.991 4.677 6.316 2
19.06 16.45 0.8854 6.416 3.719 2.248 6.163 2
18.96 16.2 0.9077 6.051 3.897 4.334 5.75 2
19.15 16.45 0.889 6.245 3.815 3.084 6.185 2
18.89 16.23 0.9008 6.227 3.769 3.639 5.966 2
20.03 16.9 0.8811 6.493 3.857 3.063 6.32 2
20.24 16.91 0.8897 6.315 3.962 5.901 6.188 2
18.14 16.12 0.8772 6.059 3.563 3.619 6.011 2
16.17 15.38 0.8588 5.762 3.387 4.286 5.703 2
18.43 15.97 0.9077 5.98 3.771 2.984 5.905 2
15.99 14.89 0.9064 5.363 3.582 3.336 5.144 2
18.75 16.18 0.8999 6.111 3.869 4.188 5.992 2
18.65 16.41 0.8698 6.285 3.594 4.391 6.102 2
17.98 15.85 0.8993 5.979 3.687 2.257 5.919 2
20.16 17.03 0.8735 6.513 3.773 1.91 6.185 2
17.55 15.66 0.8991 5.791 3.69 5.366 5.661 2
18.3 15.89 0.9108 5.979 3.755 2.837 5.962 2
18.94 16.32 0.8942 6.144 3.825 2.908 5.949 2
15.38 14.9 0.8706 5.884 3.268 4.462 5.795 2
16.16 15.33 0.8644 5.845 3.395 4.266 5.795 2
15.56 14.89 0.8823 5.776 3.408 4.972 5.847 2
15.38 14.66 0.899 5.477 3.465 3.6 5.439 2
17.36 15.76 0.8785 6.145 3.574 3.526 5.971 2
15.57 15.15 0.8527 5.92 3.231 2.64 5.879 2
15.6 15.11 0.858 5.832 3.286 2.725 5.752 2
16.23 15.18 0.885 5.872 3.472 3.769 5.922 2
13.07 13.92 0.848 5.472 2.994 5.304 5.395 3
13.32 13.94 0.8613 5.541 3.073 7.035 5.44 3
13.34 13.95 0.862 5.389 3.074 5.995 5.307 3
12.22 13.32 0.8652 5.224 2.967 5.469 5.221 3
11.82 13.4 0.8274 5.314 2.777 4.471 5.178 3
11.21 13.13 0.8167 5.279 2.687 6.169 5.275 3
11.43 13.13 0.8335 5.176 2.719 2.221 5.132 3
12.49 13.46 0.8658 5.267 2.967 4.421 5.002 3
12.7 13.71 0.8491 5.386 2.911 3.26 5.316 3
10.79 12.93 0.8107 5.317 2.648 5.462 5.194 3
11.83 13.23 0.8496 5.263 2.84 5.195 5.307 3
12.01 13.52 0.8249 5.405 2.776 6.992 5.27 3
12.26 13.6 0.8333 5.408 2.833 4.756 5.36 3
11.18 13.04 0.8266 5.22 2.693 3.332 5.001 3
11.36 13.05 0.8382 5.175 2.755 4.048 5.263 3
11.19 13.05 0.8253 5.25 2.675 5.813 5.219 3
11.34 12.87 0.8596 5.053 2.849 3.347 5.003 3
12.13 13.73 0.8081 5.394 2.745 4.825 5.22 3
11.75 13.52 0.8082 5.444 2.678 4.378 5.31 3
11.49 13.22 0.8263 5.304 2.695 5.388 5.31 3
12.54 13.67 0.8425 5.451 2.879 3.082 5.491 3
12.02 13.33 0.8503 5.35 2.81 4.271 5.308 3
12.05 13.41 0.8416 5.267 2.847 4.988 5.046 3
12.55 13.57 0.8558 5.333 2.968 4.419 5.176 3
11.14 12.79 0.8558 5.011 2.794 6.388 5.049 3
12.1 13.15 0.8793 5.105 2.941 2.201 5.056 3
12.44 13.59 0.8462 5.319 2.897 4.924 5.27 3
12.15 13.45 0.8443 5.417 2.837 3.638 5.338 3
11.35 13.12 0.8291 5.176 2.668 4.337 5.132 3
11.24 13 0.8359 5.09 2.715 3.521 5.088 3
11.02 13 0.8189 5.325 2.701 6.735 5.163 3
11.55 13.1 0.8455 5.167 2.845 6.715 4.956 3
11.27 12.97 0.8419 5.088 2.763 4.309 5 3
11.4 13.08 0.8375 5.136 2.763 5.588 5.089 3
10.83 12.96 0.8099 5.278 2.641 5.182 5.185 3
10.8 12.57 0.859 4.981 2.821 4.773 5.063 3
11.26 13.01 0.8355 5.186 2.71 5.335 5.092 3
10.74 12.73 0.8329 5.145 2.642 4.702 4.963 3
11.48 13.05 0.8473 5.18 2.758 5.876 5.002 3
12.21 13.47 0.8453 5.357 2.893 1.661 5.178 3
11.41 12.95 0.856 5.09 2.775 4.957 4.825 3
12.46 13.41 0.8706 5.236 3.017 4.987 5.147 3
12.19 13.36 0.8579 5.24 2.909 4.857 5.158 3
11.65 13.07 0.8575 5.108 2.85 5.209 5.135 3
12.89 13.77 0.8541 5.495 3.026 6.185 5.316 3
11.56 13.31 0.8198 5.363 2.683 4.062 5.182 3
11.81 13.45 0.8198 5.413 2.716 4.898 5.352 3
10.91 12.8 0.8372 5.088 2.675 4.179 4.956 3
11.23 12.82 0.8594 5.089 2.821 7.524 4.957 3
10.59 12.41 0.8648 4.899 2.787 4.975 4.794 3
10.93 12.8 0.839 5.046 2.717 5.398 5.045 3
11.27 12.86 0.8563 5.091 2.804 3.985 5.001 3
11.87 13.02 0.8795 5.132 2.953 3.597 5.132 3
10.82 12.83 0.8256 5.18 2.63 4.853 5.089 3
12.11 13.27 0.8639 5.236 2.975 4.132 5.012 3
12.8 13.47 0.886 5.16 3.126 4.873 4.914 3
12.79 13.53 0.8786 5.224 3.054 5.483 4.958 3
13.37 13.78 0.8849 5.32 3.128 4.67 5.091 3
12.62 13.67 0.8481 5.41 2.911 3.306 5.231 3
12.76 13.38 0.8964 5.073 3.155 2.828 4.83 3
12.38 13.44 0.8609 5.219 2.989 5.472 5.045 3
12.67 13.32 0.8977 4.984 3.135 2.3 4.745 3
11.18 12.72 0.868 5.009 2.81 4.051 4.828 3
12.7 13.41 0.8874 5.183 3.091 8.456 5 3
12.37 13.47 0.8567 5.204 2.96 3.919 5.001 3
12.19 13.2 0.8783 5.137 2.981 3.631 4.87 3
11.23 12.88 0.8511 5.14 2.795 4.325 5.003 3
13.2 13.66 0.8883 5.236 3.232 8.315 5.056 3
11.84 13.21 0.8521 5.175 2.836 3.598 5.044 3
12.3 13.34 0.8684 5.243 2.974 5.637 5.063 3

3
fashion_mnist.py
View File

@@ -109,8 +109,7 @@ best_score = pso_mnist.fit(
save_info=True,
log=2,
log_name="fashion_mnist",
save_path="./logs/fashion_mnist",
renewal="mse",
renewal="acc",
check_point=25,
empirical_balance=False,
dispersion=False,

12
iris.py
View File

@@ -50,22 +50,24 @@ pso_iris = optimizer(
c1=0.6,
w_min=0.5,
w_max=0.9,
negative_swarm=0.1,
negative_swarm=0,
mutation_swarm=0.2,
particle_min=-3,
particle_max=3,
convergence_reset=True,
convergence_reset_patience=10,
convergence_reset_monitor="mse",
convergence_reset_min_delta=0.05,
)
best_score = pso_iris.fit(
x_train,
y_train,
epochs=200,
epochs=500,
save_info=True,
log=2,
log_name="iris",
save_path="result/iris",
renewal="acc",
check_point=25,
validate_data=(x_test, y_test),
)
gc.collect()

44
mnist.py
View File

@@ -40,10 +40,11 @@ def make_model():
input_shape=(28, 28, 1))
)
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.5))
model.add(Conv2D(64, kernel_size=(3, 3), activation="relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dropout(0.25))
model.add(Dropout(0.5))
model.add(Dense(256, activation="relu"))
model.add(Dense(128, activation="relu"))
model.add(Dense(10, activation="softmax"))
@@ -51,23 +52,6 @@ def make_model():
return model
def random_state():
with open(
"result/mnist/20230723-061626/mean_squared_error_[0.6384999752044678, 0.0723000094294548].json",
"r",
) as f:
json_ = json.load(f)
rs = (
json_["random_state_0"],
np.array(json_["random_state_1"]),
json_["random_state_2"],
json_["random_state_3"],
json_["random_state_4"],
)
return rs
# %%
model = make_model()
x_train, y_train, x_test, y_test = get_data()
@@ -91,17 +75,17 @@ loss = [
pso_mnist = optimizer(
model,
loss="categorical_crossentropy",
n_particles=500,
n_particles=100,
c0=0.5,
c1=1.0,
w_min=0.7,
c1=0.8,
w_min=0.6,
w_max=0.9,
negative_swarm=0.05,
mutation_swarm=0.3,
negative_swarm=0.0,
mutation_swarm=0.2,
convergence_reset=True,
convergence_reset_patience=10,
convergence_reset_monitor="mse",
convergence_reset_min_delta=0.0005,
convergence_reset_monitor="loss",
convergence_reset_min_delta=0.05,
)
best_score = pso_mnist.fit(
@@ -111,13 +95,13 @@ best_score = pso_mnist.fit(
save_info=True,
log=2,
log_name="mnist",
save_path="./logs/mnist",
renewal="acc",
renewal="loss",
check_point=25,
empirical_balance=False,
empirical_balance=True,
dispersion=False,
batch_size=5000,
back_propagation=True,
batch_size=2048,
back_propagation=False,
validate_data=(x_test, y_test),
)
print("Done!")

67
mnist_tf.py
View File

@@ -39,20 +39,51 @@ def get_data_test():
return x_test, y_test
class _batch_generator:
def __init__(self, x, y, batch_size: int = 32):
self.batch_size = batch_size
class _batch_generator_:
def __init__(self, x, y, batch_size: int = None):
self.index = 0
dataset = tf.data.Dataset.from_tensor_slices((x, y))
self.dataset = list(dataset.batch(batch_size))
self.max_index = len(dataset) // batch_size
self.x = x
self.y = y
self.setBatchSize(batch_size)
def next(self):
self.index += 1
if self.index >= self.max_index:
self.index = 0
self.__getBatchSlice(self.batch_size)
return self.dataset[self.index][0], self.dataset[self.index][1]
def getMaxIndex(self):
return self.max_index
def getIndex(self):
return self.index
def setIndex(self, index):
self.index = index
def getBatchSize(self):
return self.batch_size
def setBatchSize(self, batch_size: int = None):
if batch_size is None:
batch_size = len(self.x) // 10
elif batch_size > len(self.x):
batch_size = len(self.x)
self.batch_size = batch_size
print(f"batch size : {self.batch_size}")
self.dataset = self.__getBatchSlice(self.batch_size)
self.max_index = len(self.dataset)
def __getBatchSlice(self, batch_size):
return list(
tf.data.Dataset.from_tensor_slices(
(self.x, self.y)).shuffle(len(self.x)).batch(batch_size)
)
def getDataset(self):
return self.dataset
def make_model():
model = Sequential()
@@ -60,11 +91,13 @@ def make_model():
Conv2D(32, kernel_size=(5, 5), activation="relu",
input_shape=(28, 28, 1))
)
model.add(MaxPooling2D(pool_size=(3, 3)))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.5))
model.add(Conv2D(64, kernel_size=(3, 3), activation="relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dropout(0.5))
model.add(Dense(256, activation="relu"))
model.add(Dense(128, activation="relu"))
model.add(Dense(10, activation="softmax"))
@@ -76,21 +109,25 @@ x_train, y_train, x_test, y_test = get_data()
y_train = tf.one_hot(y_train, 10)
y_test = tf.one_hot(y_test, 10)
dataset = _batch_generator(x_train, y_train, 64)
batch = 64
dataset = _batch_generator_(x_train, y_train, batch)
model.compile(optimizer="adam", loss="mse", metrics=["accuracy"])
model.compile(optimizer="adam", loss="categorical_crossentropy",
metrics=["accuracy", "mse", "mae"])
count = 0
while count < 20:
print(f"batch size : {batch}")
print("iter " + str(dataset.getMaxIndex()))
print("Training model...")
while count < dataset.getMaxIndex():
x_batch, y_batch = dataset.next()
count += 1
print("Training model...")
model.fit(x_batch, y_batch, epochs=1, batch_size=1, verbose=1)
print(f"iter {count}/{dataset.getMaxIndex()}")
model.fit(x_batch, y_batch, epochs=1, batch_size=batch, verbose=1)
print(count)
print("Evaluating model...")
model.evaluate(x_test, y_test, verbose=2)
model.evaluate(x_test, y_test, verbose=1)
weights = model.get_weights()

2
pso/__init__.py
View File

@@ -1,7 +1,7 @@
from .optimizer import Optimizer as optimizer
from .particle import Particle as particle
__version__ = "1.0.2"
__version__ = "1.0.3"
__all__ = [
"optimizer",

276
pso/optimizer.py
View File

@@ -18,6 +18,8 @@ if gpus:
except RuntimeError as r:
print(r)
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"
class Optimizer:
"""
@@ -107,12 +109,9 @@ class Optimizer:
self.w_max = w_max # 최대 관성 수치
self.negative_swarm = negative_swarm # 최적해와 반대로 이동할 파티클 비율 - 0 ~ 1 사이의 값
self.mutation_swarm = mutation_swarm # 관성을 추가로 사용할 파티클 비율 - 0 ~ 1 사이의 값
self.g_best_score = [np.inf, 0, np.inf] # 최고 점수 - 시작은 0으로 초기화
self.g_best = model.get_weights() # 최고 점수를 받은 가중치
self.avg_score = 0 # 평균 점수
# self.sigma = 1.0
self.save_path = None # 저장 위치
self.renewal = "acc"
self.dispersion = False
self.day = datetime.now().strftime("%Y%m%d-%H%M%S")
@@ -139,13 +138,12 @@ class Optimizer:
if i < self.negative_swarm * self.n_particles:
negative_count += 1
# del m, init_weights, w_, sh_, len_
gc.collect()
tf.keras.backend.reset_uids()
tf.keras.backend.clear_session()
# del model_
self.particles[0].update_global_best()
print(f"negative swarm : {negative_count} / {n_particles}")
print(f"mutation swarm : {mutation_swarm * 100}%")
@@ -171,15 +169,13 @@ class Optimizer:
del self.w_min
del self.w_max
del self.negative_swarm
del self.g_best_score
del self.g_best
del self.avg_score
gc.collect()
tf.keras.backend.reset_uids()
tf.keras.backend.clear_session()
def _encode_(self, weights):
def _encode(self, weights):
"""
가중치를 1차원으로 풀어서 반환
@@ -205,7 +201,7 @@ class Optimizer:
def _decode_(self, weight, shape, length):
"""
_encode_ 로 인코딩된 가중치를 원본 shape으로 복원
_encode 로 인코딩된 가중치를 원본 shape으로 복원
파라미터는 encode의 리턴값을 그대로 사용을 권장
Args:
@@ -243,17 +239,19 @@ class Optimizer:
"""
self.model.set_weights(weights)
score = self.model.evaluate(x, y, verbose=0)
if self.renewal == "acc":
score_ = score[1]
else:
if self.renewal == "loss":
score_ = score[0]
elif self.renewal == "acc":
score_ = score[1]
elif self.renewal == "mse":
score_ = score[2]
if score_ > 0:
return 1 / (1 + score_)
else:
return 1 + np.abs(score_)
def __weight_range__(self):
def __weight_range(self):
"""
가중치의 범위를 반환
@@ -261,7 +259,7 @@ class Optimizer:
(float): 가중치의 최소값
(float): 가중치의 최대값
"""
w_, w_s, w_l = self._encode_(self.g_best)
w_, w_s, w_l = self._encode(Particle.g_best_weights)
weight_min = np.min(w_)
weight_max = np.max(w_)
@@ -269,9 +267,8 @@ class Optimizer:
return weight_min, weight_max
class _batch_generator_:
def __init__(self, x, y, batch_size: int = 32):
self.batch_size = batch_size
class batch_generator:
def __init__(self, x, y, batch_size: int = None):
self.index = 0
self.x = x
self.y = y
@@ -281,7 +278,7 @@ class Optimizer:
self.index += 1
if self.index >= self.max_index:
self.index = 0
self.__getBatchSlice__(self.batch_size)
self.__getBatchSlice(self.batch_size)
return self.dataset[self.index][0], self.dataset[self.index][1]
def getMaxIndex(self):
@@ -296,15 +293,20 @@ class Optimizer:
def getBatchSize(self):
return self.batch_size
def setBatchSize(self, batch_size):
def setBatchSize(self, batch_size: int = None):
if batch_size is None:
batch_size = len(self.x) // 10
elif batch_size > len(self.x):
batch_size = len(self.x)
self.batch_size = batch_size
if self.batch_size > len(self.x):
self.batch_size = len(self.x)
print(f"batch size : {self.batch_size}")
self.dataset = self.__getBatchSlice__(self.batch_size)
self.max_index = len(self.dataset)
def __getBatchSlice__(self, batch_size):
print(f"batch size : {self.batch_size}")
self.dataset = self.__getBatchSlice(self.batch_size)
self.max_index = len(self.dataset)
if batch_size % len(self.x) != 0:
self.max_index -= 1
def __getBatchSlice(self, batch_size):
return list(
tf.data.Dataset.from_tensor_slices(
(self.x, self.y)).shuffle(len(self.x)).batch(batch_size)
@@ -321,13 +323,12 @@ class Optimizer:
log: int = 0,
log_name: str = None,
save_info: bool = False,
save_path: str = "./logs",
renewal: str = "mse",
empirical_balance: bool = False,
dispersion: bool = False,
check_point: int = None,
batch_size: int = None,
validate_data: any = None,
validate_data: tuple = None,
back_propagation: bool = False,
):
"""
@@ -344,6 +345,8 @@ class Optimizer:
check_point : int - 저장할 위치 - None : 저장 안함
batch_size : int - batch size default : None => len(x) // 10
batch_size > len(x) : auto max batch size
validate_data : tuple - (x, y) default : None => (x, y)
back_propagation : bool - True : back propagation, False : not back propagation
"""
try:
if x.shape[0] != y.shape[0]:
@@ -352,47 +355,53 @@ class Optimizer:
if log not in [0, 1, 2]:
raise ValueError("log not in [0, 1, 2]")
if save_info and save_path is None:
raise ValueError("save_path is None")
if renewal not in ["acc", "loss", "mse"]:
raise ValueError("renewal not in ['acc', 'loss', 'mse']")
if check_point is not None and save_path is None:
raise ValueError("save_path is None")
if validate_data is not None:
if validate_data[0].shape[0] != validate_data[1].shape[0]:
raise ValueError("validate_data shape error")
if validate_data is None:
validate_data = (x, y)
if batch_size is not None and batch_size < 1:
raise ValueError("batch_size < 1")
if batch_size is None or batch_size > len(x):
batch_size = len(x)
except ValueError as ve:
sys.exit(ve)
except Exception as e:
sys.exit(e)
self.save_path = save_path
self.empirical_balance = empirical_balance
self.dispersion = dispersion
if batch_size is None:
batch_size = len(x) // 10
self.renewal = renewal
particle_sum = 0 # x_j
try:
train_log_dir = "logs/fit/" + self.day
if log_name is None:
log_name = "fit"
self.log_path = f"logs/{log_name}/{self.day}"
if log == 2:
assert log_name is not None, "log_name is None"
train_log_dir = f"logs/{log_name}/{self.day}/train"
train_log_dir = self.log_path + "/train"
for i in range(self.n_particles):
self.train_summary_writer[i] = tf.summary.create_file_writer(
train_log_dir + f"/{i}"
)
elif check_point is not None or log == 1:
if save_path is None:
raise ValueError("save_path is None")
else:
self.save_path = save_path
if not os.path.exists(f"{save_path}/{self.day}"):
os.makedirs(f"{save_path}/{self.day}", exist_ok=True)
if not os.path.exists(self.log_path):
os.makedirs(self.log_path, exist_ok=True)
except ValueError as ve:
sys.exit(ve)
except Exception as e:
sys.exit(e)
if back_propagation:
model_ = keras.models.model_from_json(self.model.to_json())
@@ -404,19 +413,19 @@ class Optimizer:
model_.fit(x, y, epochs=1, verbose=0)
score = model_.evaluate(x, y, verbose=1)
self.g_best_score = score
Particle.g_best_score = score
self.g_best = model_.get_weights()
Particle.g_best_weights = model_.get_weights()
del model_
dataset = self._batch_generator_(x, y, batch_size=batch_size)
dataset = self.batch_generator(x, y, batch_size=batch_size)
try:
epoch_sum = 0
epochs_pbar = tqdm(
range(epochs),
desc=f"best - loss: {self.g_best_score[0]:.4f} - acc: {self.g_best_score[1]:.4f} - mse: {self.g_best_score[2]:.4f}",
desc=f"best - loss: {Particle.g_best_score[0]:.4f} - acc: {Particle.g_best_score[1]:.4f} - mse: {Particle.g_best_score[2]:.4f}",
ascii=True,
leave=True,
position=0,
@@ -429,6 +438,8 @@ class Optimizer:
max_acc = 0
min_loss = np.inf
min_mse = np.inf
# 한번의 실행 동안 최고 점수를 받은 파티클의 인덱스
best_particle_index = 0
# epoch_particle_sum = 0
part_pbar = tqdm(
range(len(self.particles)),
@@ -437,33 +448,34 @@ class Optimizer:
leave=False,
position=1,
)
w = self.w_max - (self.w_max - self.w_min) * epoch / epochs
for i in part_pbar:
part_pbar.set_description(
f"loss: {min_loss:.4f} acc: {max_acc:.4f} mse: {min_mse:.4f}"
)
g_best = self.g_best
g_best = Particle.g_best_weights
x_batch, y_batch = dataset.next()
weight_min, weight_max = self.__weight_range__()
weight_min, weight_max = self.__weight_range()
if dispersion:
ts = weight_min + np.random.rand() * (weight_max - weight_min)
g_, g_sh, g_len = self._encode_(self.g_best)
g_, g_sh, g_len = self._encode(
Particle.g_best_weights)
decrement = (epochs - epoch + 1) / epochs
g_ = (1 - decrement) * g_ + decrement * ts
g_best = self._decode(g_, g_sh, g_len)
g_best = self._decode_(g_, g_sh, g_len)
if empirical_balance:
if np.random.rand() < np.exp(-(epoch) / epochs):
w_p_ = self._f(
x, y, self.particles[i].get_best_weights()
x_batch, y_batch, self.particles[i].get_best_weights(
)
w_g_ = self._f(x, y, g_best)
)
w_g_ = self._f(x_batch, y_batch, g_best)
w_p = w_p_ / (w_p_ + w_g_)
w_g = w_p_ / (w_p_ + w_g_)
@@ -473,7 +485,7 @@ class Optimizer:
else:
p_b = self.particles[i].get_best_score()
g_a = self.avg_score
l_b = p_b - g_a
l_b = p_b[1] - g_a
sigma_post = np.sqrt(np.power(l_b, 2))
sigma_pre = (
1
@@ -517,7 +529,6 @@ class Optimizer:
self.c0,
self.c1,
w,
g_best,
w_p,
w_g,
renewal=renewal,
@@ -526,7 +537,7 @@ class Optimizer:
else:
score = self.particles[i].step(
x_batch, y_batch, self.c0, self.c1, w, g_best, renewal=renewal
x_batch, y_batch, self.c0, self.c1, w, renewal=renewal
)
if log == 2:
@@ -537,98 +548,84 @@ class Optimizer:
)
tf.summary.scalar("mse", score[2], step=epoch + 1)
if renewal == "acc":
# 최고 점수 보다 높거나 같을 경우
if score[1] >= max_acc:
# 각 점수 갱신
min_loss, max_acc, min_mse = score
# 최고 점수 보다 같거나 높을 경우
if score[1] >= self.g_best_score[1]:
# 최고 점수 보다 높을 경우
if score[1] > self.g_best_score[1]:
# 최고 점수 갱신
self.g_best_score = score
# 최고 weight 갱신
self.g_best = self.particles[i].get_best_weights(
)
# 최고 점수 와 같을 경우
else:
# 최저 loss 보다 낮을 경우
if score[0] < self.g_best_score[0]:
self.g_best_score[0] = score[0]
self.g_best = self.particles[i].get_best_weights(
)
epochs_pbar.set_description(
f"best - loss: {self.g_best_score[0]:.4f} - acc: {self.g_best_score[1]:.4f} - mse: {self.g_best_score[2]:.4f}"
)
elif renewal == "loss":
if renewal == "loss":
# 최저 loss 보다 작거나 같을 경우
if score[0] <= min_loss:
if score[0] < min_loss:
# 각 점수 갱신
min_loss, max_acc, min_mse = score
# 최저 loss 와 같거나 작을 경우
if score[0] <= self.g_best_score[0]:
# 최저 loss 보다 작을 경우
if score[0] < self.g_best_score[0]:
# 최고 점수 갱신
self.g_best_score = score
# 최고 weight 갱신
self.g_best = self.particles[i].get_best_weights(
)
# 최저 loss 와 같을 경우
else:
# 최고 acc 보다 높을 경우
if score[1] > self.g_best_score[1]:
self.g_best_score[1] = score[1]
self.g_best = self.particles[i].get_best_weights(
)
epochs_pbar.set_description(
f"best - loss: {self.g_best_score[0]:.4f} - acc: {self.g_best_score[1]:.4f} - mse: {self.g_best_score[2]:.4f}"
)
best_particle_index = i
elif renewal == "acc":
# 최고 점수 보다 높거나 같을 경우
if score[1] > max_acc:
# 각 점수 갱신
min_loss, max_acc, min_mse = score
best_particle_index = i
elif renewal == "mse":
if score[2] <= min_mse:
if score[2] < min_mse:
min_loss, max_acc, min_mse = score
if score[2] <= self.g_best_score[2]:
if score[2] < self.g_best_score[2]:
self.g_best_score[0] = score[0]
self.g_best_score[1] = score[1]
self.g_best_score[2] = score[2]
self.g_best = self.particles[i].get_best_weights(
)
else:
if score[1] > self.g_best_score[1]:
self.g_best_score[1] = score[1]
self.g_best = self.particles[i].get_best_weights(
)
epochs_pbar.set_description(
f"best - loss: {self.g_best_score[0]:.4f} - acc: {self.g_best_score[1]:.4f} - mse: {self.g_best_score[2]:.4f}"
)
best_particle_index = i
particle_sum += score[1]
if log == 1:
with open(
f"./{save_path}/{self.day}/{self.n_particles}_{epochs}_{self.c0}_{self.c1}_{self.w_min}_{renewal}.csv",
f"./logs/{log_name}/{self.day}/{self.n_particles}_{epochs}_{self.c0}_{self.c1}_{self.w_min}_{renewal}.csv",
"a",
) as f:
f.write(f"{score[0]}, {score[1]}")
f.write(f"{score[0]}, {score[1]}, {score[2]}")
if i != self.n_particles - 1:
f.write(", ")
else:
f.write("\n")
part_pbar.refresh()
# 한번 epoch 가 끝나고 갱신을 진행해야 순간적으로 높은 파티클이 발생해도 오류가 생기지 않음
if renewal == "loss":
if min_loss <= Particle.g_best_score[0]:
if min_loss < Particle.g_best_score[0]:
self.particles[best_particle_index].update_global_best(
)
else:
if max_acc > Particle.g_best_score[1]:
self.particles[best_particle_index].update_global_best(
)
elif renewal == "acc":
if max_acc >= Particle.g_best_score[1]:
# 최고 점수 보다 높을 경우
if max_acc > Particle.g_best_score[1]:
# 최고 점수 갱신
self.particles[best_particle_index].update_global_best(
)
# 최고 점수 와 같을 경우
else:
# 최저 loss 보다 낮을 경우
if min_loss < Particle.g_best_score[0]:
self.particles[best_particle_index].update_global_best(
)
elif renewal == "mse":
if min_mse <= Particle.g_best_score[2]:
if min_mse < Particle.g_best_score[2]:
self.particles[best_particle_index].update_global_best(
)
else:
if max_acc > Particle.g_best_score[1]:
self.particles[best_particle_index].update_global_best(
)
# 최고 점수 갱신
epochs_pbar.set_description(
f"best - loss: {Particle.g_best_score[0]:.4f} - acc: {Particle.g_best_score[1]:.4f} - mse: {Particle.g_best_score[2]:.4f}"
)
if check_point is not None:
if epoch % check_point == 0:
os.makedirs(f"./{save_path}/{self.day}", exist_ok=True)
os.makedirs(
f"./logs/{log_name}/{self.day}", exist_ok=True)
self._check_point_save(
f"./{save_path}/{self.day}/ckpt-{epoch}")
f"./logs/{log_name}/{self.day}/ckpt-{epoch}")
tf.keras.backend.reset_uids()
tf.keras.backend.clear_session()
@@ -644,13 +641,13 @@ class Optimizer:
print(e)
finally:
self.model_save(x, y, save_path)
self.model_save(validate_data)
print("model save")
if save_info:
self.save_info(save_path)
self.save_info()
print("save info")
return self.g_best_score
return Particle.g_best_score
def get_best_model(self):
"""
@@ -660,7 +657,7 @@ class Optimizer:
(keras.models): 모델
"""
model = keras.models.model_from_json(self.model.to_json())
model.set_weights(self.g_best)
model.set_weights(Particle.g_best_weights)
model.compile(
loss=self.loss,
optimizer="adam",
@@ -676,7 +673,7 @@ class Optimizer:
Returns:
(float): 점수
"""
return self.g_best_score
return Particle.g_best_score
def get_best_weights(self):
"""
@@ -685,9 +682,9 @@ class Optimizer:
Returns:
(float): 가중치
"""
return self.g_best
return Particle.g_best_weights
def save_info(self, path: str = "./result"):
def save_info(self):
"""
학습 정보를 저장
@@ -697,7 +694,7 @@ class Optimizer:
json_save = {
"name": f"{self.day}/{self.n_particles}_{self.c0}_{self.c1}_{self.w_min}.h5",
"n_particles": self.n_particles,
"score": self.g_best_score,
"score": Particle.g_best_score,
"c0": self.c0,
"c1": self.c1,
"w_min": self.w_min,
@@ -716,7 +713,7 @@ class Optimizer:
}
with open(
f"./{path}/{self.day}/{self.loss}_{self.g_best_score}.json",
f"./{self.log_path}/{self.loss}_{Particle.g_best_score}.json",
"a",
) as f:
json.dump(json_save, f, indent=4)
@@ -731,7 +728,7 @@ class Optimizer:
model = self.get_best_model()
model.save_weights(save_path)
def model_save(self, x, y, save_path: str = "./result"):
def model_save(self, valid_data: tuple = None):
"""
최고 점수를 받은 모델 저장
@@ -741,11 +738,12 @@ class Optimizer:
Returns:
(keras.models): 모델
"""
x, y = valid_data
model = self.get_best_model()
score = model.evaluate(x, y, verbose=1)
print(
f"model score - loss: {score[0]} - acc: {score[1]} - mse: {score[2]}")
model.save(
f"./{save_path}/{self.day}/model_{score[0 if self.renewal == 'loss' else 1 if self.renewal == 'acc' else 2 ]}.h5"
f"./{self.log_path}/model_{score[0 if self.renewal == 'loss' else 1 if self.renewal == 'acc' else 2 ]}.h5"
)
return model

126
pso/particle.py
View File

@@ -1,5 +1,4 @@
import numpy as np
from tensorflow import keras
@@ -13,11 +12,14 @@ class Particle:
4. 가중치 업데이트
5. 2번으로 돌아가서 반복
"""
g_best_score = [np.inf, 0, np.inf]
g_best_weights = None
count = 0
def __init__(
self,
model: keras.models,
loss,
loss: any = None,
negative: bool = False,
mutation: float = 0,
converge_reset: bool = False,
@@ -50,7 +52,7 @@ class Particle:
print(e)
exit(1)
self.__reset_particle__()
self.__reset_particle()
self.best_weights = self.model.get_weights()
self.before_best = self.model.get_weights()
self.negative = negative
@@ -62,6 +64,7 @@ class Particle:
self.converge_reset_patience = converge_reset_patience
self.converge_reset_monitor = converge_reset_monitor
self.converge_reset_min_delta = converge_reset_min_delta
Particle.count += 1
def __del__(self):
del self.model
@@ -70,6 +73,7 @@ class Particle:
del self.negative
del self.best_score
del self.best_weights
Particle.count -= 1
def _encode(self, weights: list):
"""
@@ -135,35 +139,37 @@ class Particle:
score = self.model.evaluate(x, y, verbose=0)
if renewal == "loss":
if score[0] < self.best_score[0]:
self.best_score[0] = score[0]
self.best_score = score
self.best_weights = self.model.get_weights()
elif renewal == "acc":
if score[1] > self.best_score[1]:
self.best_score[1] = score[1]
self.best_score = score
self.best_weights = self.model.get_weights()
elif renewal == "mse":
if score[2] < self.best_score[2]:
self.best_score[2] = score[2]
self.best_score = score
self.best_weights = self.model.get_weights()
else:
raise ValueError("renewal must be 'acc' or 'loss' or 'mse'")
return score
def __check_converge_reset__(self, score, monitor="loss", patience: int = 10, min_delta: float = 0.0001):
def __check_converge_reset(self, score, monitor: str = None, patience: int = 10, min_delta: float = 0.0001):
"""
early stop을 구현한 함수
Args:
score (float): 현재 점수 [0] - loss, [1] - acc
monitor (str, optional): 감시할 점수. Defaults to "loss".
monitor (str, optional): 감시할 점수. Defaults to acc. | "acc" or "loss" or "mse"
patience (int, optional): early stop을 위한 기다리는 횟수. Defaults to 10.
min_delta (float, optional): early stop을 위한 최소 변화량. Defaults to 0.0001.
"""
if monitor in ["acc", "accuracy"]:
self.score_history.append(score[1])
elif monitor in ["loss"]:
if monitor is None:
monitor = "acc"
if monitor in ["loss"]:
self.score_history.append(score[0])
elif monitor in ["acc", "accuracy"]:
self.score_history.append(score[1])
elif monitor in ["mse"]:
self.score_history.append(score[2])
else:
@@ -176,7 +182,7 @@ class Particle:
return True
return False
def __reset_particle__(self):
def __reset_particle(self):
self.model = keras.models.model_from_json(self.model.to_json())
self.model.compile(
optimizer="adam",
@@ -184,13 +190,13 @@ class Particle:
metrics=["accuracy", "mse"]
)
i_w_, i_s, i_l = self._encode(self.model.get_weights())
i_w_ = np.random.uniform(-0.05, 0.05, len(i_w_))
i_w_ = np.random.uniform(-0.05, 0.1, len(i_w_))
self.velocities = self._decode(i_w_, i_s, i_l)
del i_w_, i_s, i_l
self.score_history = []
def _update_velocity(self, local_rate, global_rate, w, g_best):
def _update_velocity(self, local_rate, global_rate, w):
"""
현재 속도 업데이트
@@ -198,19 +204,20 @@ class Particle:
local_rate (float): 지역 최적해의 영향력
global_rate (float): 전역 최적해의 영향력
w (float): 현재 속도의 영향력 - 관성 | 0.9 ~ 0.4 이 적당
g_best (list): 전역 최적해
"""
encode_w, w_sh, w_len = self._encode(weights=self.model.get_weights())
encode_v, v_sh, v_len = self._encode(weights=self.velocities)
encode_p, p_sh, p_len = self._encode(weights=self.best_weights)
encode_g, g_sh, g_len = self._encode(weights=g_best)
encode_g, g_sh, g_len = self._encode(weights=Particle.g_best_weights)
encode_before, before_sh, before_len = self._encode(
weights=self.before_best
)
r_0 = np.random.rand()
r_1 = np.random.rand()
# 이전 전역 최적해와 현재 전역 최적해가 다르면 관성을 순간적으로 증가 - 값이 바뀔 경우 기존 관성을 특정 기간동안 유지
if not np.array_equal(encode_before, encode_g, equal_nan=True):
# 이전 가중치 중요도의 1.5 배로 관성을 증가
self.before_w = w * 0.5
w = w + self.before_w
else:
@@ -218,13 +225,14 @@ class Particle:
w = w + self.before_w
if self.negative:
# 지역 최적해와 전역 최적해를 음수로 사용하여 전역 탐색을 유도
new_v = (
w * encode_v
+ local_rate * r_0 * (encode_p - encode_w)
+ -1 * global_rate * r_1 * (encode_g - encode_w)
- local_rate * r_0 * (encode_p - encode_w)
- global_rate * r_1 * (encode_g - encode_w)
)
if len(self.score_history) > 10 and max(self.score_history[-10:]) - min(self.score_history[-10:]) < 0.01:
self.__reset_particle__()
self.__reset_particle()
else:
new_v = (
@@ -246,7 +254,7 @@ class Particle:
del encode_before, before_sh, before_len
del r_0, r_1
def _update_velocity_w(self, local_rate, global_rate, w, w_p, w_g, g_best):
def _update_velocity_w(self, local_rate, global_rate, w, w_p, w_g):
"""
현재 속도 업데이트
기본 업데이트의 변형으로 지역 최적해와 전역 최적해를 분산시켜 조기 수렴을 방지
@@ -257,12 +265,11 @@ class Particle:
w (float): 현재 속도의 영향력 - 관성 | 0.9 ~ 0.4 이 적당
w_p (float): 지역 최적해의 분산 정도
w_g (float): 전역 최적해의 분산 정도
g_best (list): 전역 최적해
"""
encode_w, w_sh, w_len = self._encode(weights=self.model.get_weights())
encode_v, v_sh, v_len = self._encode(weights=self.velocities)
encode_p, p_sh, p_len = self._encode(weights=self.best_weights)
encode_g, g_sh, g_len = self._encode(weights=g_best)
encode_g, g_sh, g_len = self._encode(weights=Particle.g_best_weights)
encode_before, before_sh, before_len = self._encode(
weights=self.before_best
)
@@ -275,11 +282,12 @@ class Particle:
else:
self.before_w *= 0.75
w = w + self.before_w
if self.negative:
new_v = (
w * encode_v
+ local_rate * r_0 * (w_p * encode_p - encode_w)
+ -1 * global_rate * r_1 * (w_g * encode_g - encode_w)
- local_rate * r_0 * (w_p * encode_p - encode_w)
- global_rate * r_1 * (w_g * encode_g - encode_w)
)
else:
new_v = (
@@ -289,7 +297,7 @@ class Particle:
)
if np.random.rand() < self.mutation:
m_v = np.random.uniform(-0.1, 0.1, len(encode_v))
m_v = np.random.uniform(-0.05, 0.05, len(encode_v))
new_v = m_v
self.velocities = self._decode(new_v, w_sh, w_len)
@@ -313,7 +321,7 @@ class Particle:
del encode_w, w_sh, w_len
del encode_v, v_sh, v_len
def step(self, x, y, local_rate, global_rate, w, g_best, renewal: str = "acc"):
def step(self, x, y, local_rate, global_rate, w, renewal: str = "acc"):
"""
파티클의 한 스텝을 진행합니다.
@@ -329,19 +337,35 @@ class Particle:
Returns:
list: 현재 파티클의 점수
"""
self._update_velocity(local_rate, global_rate, w, g_best)
self._update_velocity(local_rate, global_rate, w)
self._update_weights()
score = self.get_score(x, y, renewal)
if self.converge_reset and self.__check_converge_reset__(
if self.converge_reset and self.__check_converge_reset(
score, self.converge_reset_monitor, self.converge_reset_patience, self.converge_reset_min_delta):
self.__reset_particle__()
self.__reset_particle()
score = self.get_score(x, y, renewal)
while np.isnan(score[0]) or np.isnan(score[1]) or np.isnan(score[2]) or score[0] == 0 or score[1] == 0 or score[2] == 0 or np.isinf(score[0]) or np.isinf(score[1]) or np.isinf(score[2]) or score[0] > 1000 or score[1] > 1 or score[2] > 1000:
self.__reset_particle()
score = self.get_score(x, y, renewal)
# # score 가 inf 이면 가중치를 초기화
# # score 가 nan 이면 가중치를 초기화
# # score 가 0 이면 가중치를 초기화
# if np.isinf(score[0]) or np.isinf(score[1]) or np.isinf(score[2]) or np.isnan(score[0]) or np.isnan(score[1]) or np.isnan(score[2]) or score[0] == 0 or score[1] == 0 or score[2] == 0:
# self.__reset_particle()
# score = self.get_score(x, y, renewal)
# # score 가 상식적인 범위를 벗어나면 가중치를 초기화
# if score[0] > 1000 or score[1] > 1 or score[2] > 1000:
# self.__reset_particle()
# score = self.get_score(x, y, renewal)
return score
def step_w(
self, x, y, local_rate, global_rate, w, g_best, w_p, w_g, renewal: str = "acc"
self, x, y, local_rate, global_rate, w, w_p, w_g, renewal: str = "acc"
):
"""
파티클의 한 스텝을 진행합니다.
@@ -361,10 +385,32 @@ class Particle:
Returns:
float: 현재 파티클의 점수
"""
self._update_velocity_w(local_rate, global_rate, w, w_p, w_g, g_best)
self._update_velocity_w(local_rate, global_rate, w, w_p, w_g)
self._update_weights()
return self.get_score(x, y, renewal)
score = self.get_score(x, y, renewal)
if self.converge_reset and self.__check_converge_reset(
score, self.converge_reset_monitor, self.converge_reset_patience, self.converge_reset_min_delta):
self.__reset_particle()
score = self.get_score(x, y, renewal)
while np.isnan(score[0]) or np.isnan(score[1]) or np.isnan(score[2]) or score[0] == 0 or score[1] == 0 or score[2] == 0 or np.isinf(score[0]) or np.isinf(score[1]) or np.isinf(score[2]) or score[0] > 1000 or score[1] > 1 or score[2] > 1000:
self.__reset_particle()
score = self.get_score(x, y, renewal)
# # score 가 inf 이면 가중치를 초기화
# # score 가 nan 이면 가중치를 초기화
# # score 가 0 이면 가중치를 초기화
# if np.isinf(score[0]) or np.isinf(score[1]) or np.isinf(score[2]) or np.isnan(score[0]) or np.isnan(score[1]) or np.isnan(score[2]) or score[0] == 0 or score[1] == 0 or score[2] == 0:
# self.__reset_particle()
# score = self.get_score(x, y, renewal)
# # score 가 상식적인 범위를 벗어나면 가중치를 초기화
# if score[0] > 1000 or score[1] > 1 or score[2] > 1000:
# self.__reset_particle()
# score = self.get_score(x, y, renewal)
return score
def get_best_score(self):
"""
@@ -383,3 +429,19 @@ class Particle:
list: 가중치 리스트
"""
return self.best_weights
def set_global_score(self):
"""전역 최고점수를 현재 파티클의 최고점수로 설정합니다
"""
Particle.g_best_score = self.best_score
def set_global_weights(self):
"""전역 최고점수를 받은 가중치를 현재 파티클의 최고점수를 받은 가중치로 설정합니다
"""
Particle.g_best_weights = self.best_weights
def update_global_best(self):
"""현재 파티클의 점수와 가중치를 전역 최고점수와 가중치로 설정합니다
"""
self.set_global_score()
self.set_global_weights()

112
seeds.py Normal file
View File

@@ -0,0 +1,112 @@
# %%
import json
import os
import sys
import numpy as np
import pandas as pd
import tensorflow as tf
from keras.layers import Dense
from keras.models import Sequential
from keras.utils import to_categorical
from sklearn.model_selection import train_test_split
from tensorflow import keras
from pso import optimizer
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"
def get_data():
with open("data/seeds/seeds_dataset.txt", "r", encoding="utf-8") as f:
data = f.readlines()
df = pd.DataFrame([d.split() for d in data])
df.columns = [
"area",
"perimeter",
"compactness",
"length_of_kernel",
"width_of_kernel",
"asymmetry_coefficient",
"length_of_kernel_groove",
"target",
]
df = df.astype(float)
df["target"] = df["target"].astype(int)
x = df.iloc[:, :-1].values.round(0).astype(int)
y = df.iloc[:, -1].values
y_class = to_categorical(y)
x_train, x_test, y_train, y_test = train_test_split(
x, y_class, test_size=0.2, shuffle=True
)
return x_train, y_train, x_test, y_test
def make_model():
model = Sequential()
model.add(Dense(16, activation="relu", input_shape=(7,)))
model.add(Dense(32, activation="relu"))
model.add(Dense(4, activation="softmax"))
return model
# %%
model = make_model()
x_train, y_train, x_test, y_test = get_data()
loss = [
"mean_squared_error",
"categorical_crossentropy",
"sparse_categorical_crossentropy",
"binary_crossentropy",
"kullback_leibler_divergence",
"poisson",
"cosine_similarity",
"log_cosh",
"huber_loss",
"mean_absolute_error",
"mean_absolute_percentage_error",
]
# rs = random_state()
pso_mnist = optimizer(
model,
loss="categorical_crossentropy",
n_particles=100,
c0=0.5,
c1=1.0,
w_min=0.7,
w_max=1.2,
negative_swarm=0.0,
mutation_swarm=0.3,
convergence_reset=True,
convergence_reset_patience=10,
convergence_reset_monitor="mse",
convergence_reset_min_delta=0.0005,
)
best_score = pso_mnist.fit(
x_train,
y_train,
epochs=500,
save_info=True,
log=2,
log_name="seeds",
renewal="acc",
check_point=25,
empirical_balance=False,
dispersion=False,
back_propagation=False,
validate_data=(x_test, y_test),
)
print("Done!")
sys.exit(0)

1
xor.py
View File

@@ -67,7 +67,6 @@ best_score = pso_xor.fit(
save_info=True,
log=2,
log_name="xor",
save_path="./result/xor",
renewal="acc",
check_point=25,
)