Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X..........................XX.............................] duration:
46.57660645s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 16:03:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 46.576623502s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X................X.........................................] duration:
8.434559976s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 16:03:09 UTC)
orbiter warning: context deadline exceeded
Duration: 8.434585915s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [X..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XXXXXXXXXXXXXX.XXXXXXXXX] duration:
5.001506227s trace:
https://jaeger.dev.storj.io/trace/3a368df32f4e4ab4
FAIL (2025-12-29 16:03:08 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00152898s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.XXX..XX.X..X..XX.XX...X.X..XX....XX.......X....X.X......X..] duration:
6.74388218s trace:
https://jaeger.dev.storj.io/trace/713470deadd5c765
FAIL (2025-12-29 16:03:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: HSLAMA5PKLMEWPU43O432WDUJBX4ITOW52I4HFA573PAZASQIVPA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 2UORUTXJTNKR4ORJPUFH5LTRKN62ZYERE7R4OKM6U3CN2BMR7HGA) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: NMXHKF2WVPG7KF7OM54A5WUUZXT33V5VOUDAH4MV6FUJYLEITSPQ) context deadline exceeded
======LOGS======
Dec 29 16:00:59.061246935 (576h31m58.93232997s) [upload=505421]: Starting upload
Dec 29 16:00:59.061375279 (576h31m58.932458324s) [upload=505421]: Done starting upload
Dec 29 16:00:59.061477925 (576h31m58.93256097s) [upload=505421]: Uploading segments...
Dec 29 16:00:59.061577605 (576h31m58.93266065s) [upload=505421]: Got next segment. Inline: false
Dec 29 16:00:59.061611459 (576h31m58.932694494s) [upload=505421]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 16:00:59.061616889 (576h31m58.932699934s) [upload=505421]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 16:00:59.308224802 (576h31m59.179307837s) [seg_pos={0 0}, upload=505421]: Begin upload segment...
Dec 29 16:00:59.308278834 (576h31m59.179361879s) [seg_pos={0 0}, upload=505421]: Done begin upload segment.
Dec 29 16:00:59.308315033 (576h31m59.179398078s) [noise=true , address=145.79.227.16:4910 , piece_id=6TUL54UVPKW42PT24ANUSMZFCCN56LOHPKNPKNL32SKMWPGOMC3Q, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.308355600 (576h31m59.179438645s) [noise=true , address=145.79.227.18:4007 , piece_id=ZTEJZFB7LA7D22XCZSVSEWW7NMQGBVYVLWYDTMKRUIQSRNEBM7OA, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.308865201 (576h31m59.179948246s) [noise=true , address=145.79.227.2:4113 , piece_id=SB6LSZ4RVM2UR5JPAAF7R4CMU6PDJGULYJBQILALRTOIQDCUASRQ, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.308874338 (576h31m59.179957383s) [noise=true , address=77.90.31.21:4878 , piece_id=Y62OMBKCLT6ASX7YY6DIAGXCEMHHJ5TGJGQ5MAJ4JTXN3K5ETN4Q, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.309405410 (576h31m59.180488455s) [noise=true , address=45.91.116.9:4264 , piece_id=2UORUTXJTNKR4ORJPUFH5LTRKN62ZYERE7R4OKM6U3CN2BMR7HGA, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.309549134 (576h31m59.180632179s) [noise=true , address=77.90.31.13:4629 , piece_id=NMXHKF2WVPG7KF7OM54A5WUUZXT33V5VOUDAH4MV6FUJYLEITSPQ, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.309850056 (576h31m59.180933101s) [noise=true , address=145.79.227.32:4836 , piece_id=4WWQUDGPLIQSGMGIFMBDQWGTIU7Z6BAJ6MFX5BNPXRTFQ36WYTJQ, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.310054766 (576h31m59.181137811s) [noise=true , address=145.79.227.14:4837 , piece_id=SV7S4CG4QLIJTCZHET2JA2RX5TU6LGF6XZTEIIXG436ULYNZANSQ, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.310324079 (576h31m59.181407134s) [noise=true , address=145.79.227.38:4978 , piece_id=E7JGEIVTOFC6NB7WILMH3EHKFBMS6YIQFMSL57MOFQMVR67D2VKA, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.310621887 (576h31m59.181704932s) [noise=true , address=145.79.227.33:4871 , piece_id=USISPRAQJYRMKRCFG4I2SAUJRKW4WLV5DRXPY4TF3W2XIY6QJXNQ, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.310818741 (576h31m59.181901786s) [noise=true , address=45.91.116.14:4191 , piece_id=EMSGCRYFMI7LOUOT3LFQP4EIXY2O4NDU3RTCVEQ3LGH2SXKMQYQA, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.311091601 (576h31m59.182174646s) [noise=true , address=77.90.31.11:4554 , piece_id=TAABWVKM6E35CB332R4UZS47TWFULXLWJK67MPD6WZNXRTGJEHYA, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.311223231 (576h31m59.182306276s) [noise=true , address=145.79.227.28:4726 , piece_id=3FXO4JNTT3DYATMLLXYHYH33NESEMQ7FOY4SKK4WT37BLVNDSACQ, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.311573268 (576h31m59.182656313s) [noise=true , address=45.91.116.10:4083 , piece_id=IUPXAWEMC6NZ2BU62UUNXFJMLYAZDF46ZOPFQU2TCGN2UNQ7BUDA, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.311638462 (576h31m59.182721497s) [noise=true , address=145.79.227.6:4259 , piece_id=CZTZBSLSWG7ZGB4YOLCE67SDDW2J56FUATQ7AYKKXFHNSQG5H7EQ, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.312460958 (576h31m59.183544003s) [noise=true , address=45.91.116.12:4226 , piece_id=HSLAMA5PKLMEWPU43O432WDUJBX4ITOW52I4HFA573PAZASQIVPA, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:01:01.000597506 (576h32m0.871680572s) [upload=505421]: Next returned nil segment
Dec 29 16:01:01.000621773 (576h32m0.871704838s) [upload=505421]: Waiting for error group managing segments...
Dec 29 16:01:01.286374995 (576h32m1.15745804s) [noise=true , address=145.79.227.2:4113 , piece_id=SB6LSZ4RVM2UR5JPAAF7R4CMU6PDJGULYJBQILALRTOIQDCUASRQ, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.299456593 (576h32m1.170539639s) [noise=true , address=145.79.227.18:4007 , piece_id=ZTEJZFB7LA7D22XCZSVSEWW7NMQGBVYVLWYDTMKRUIQSRNEBM7OA, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.299739893 (576h32m1.170822928s) [noise=true , address=145.79.227.28:4726 , piece_id=3FXO4JNTT3DYATMLLXYHYH33NESEMQ7FOY4SKK4WT37BLVNDSACQ, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.301403240 (576h32m1.172486285s) [noise=true , address=145.79.227.32:4836 , piece_id=4WWQUDGPLIQSGMGIFMBDQWGTIU7Z6BAJ6MFX5BNPXRTFQ36WYTJQ, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.310564936 (576h32m1.181647981s) [noise=true , address=145.79.227.6:4259 , piece_id=CZTZBSLSWG7ZGB4YOLCE67SDDW2J56FUATQ7AYKKXFHNSQG5H7EQ, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.319403225 (576h32m1.19048627s) [noise=true , address=145.79.227.14:4837 , piece_id=SV7S4CG4QLIJTCZHET2JA2RX5TU6LGF6XZTEIIXG436ULYNZANSQ, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.321985914 (576h32m1.193068959s) [noise=true , address=145.79.227.16:4910 , piece_id=6TUL54UVPKW42PT24ANUSMZFCCN56LOHPKNPKNL32SKMWPGOMC3Q, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.324207393 (576h32m1.195290438s) [noise=true , address=145.79.227.38:4978 , piece_id=E7JGEIVTOFC6NB7WILMH3EHKFBMS6YIQFMSL57MOFQMVR67D2VKA, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.505165477 (576h32m1.376248522s) [noise=true , address=77.90.31.13:4629 , piece_id=NMXHKF2WVPG7KF7OM54A5WUUZXT33V5VOUDAH4MV6FUJYLEITSPQ, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.581245907 (576h32m1.452328952s) [noise=true , address=77.90.31.21:4878 , piece_id=Y62OMBKCLT6ASX7YY6DIAGXCEMHHJ5TGJGQ5MAJ4JTXN3K5ETN4Q, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.615014799 (576h32m1.486097844s) [noise=true , address=45.91.116.12:4226 , piece_id=HSLAMA5PKLMEWPU43O432WDUJBX4ITOW52I4HFA573PAZASQIVPA, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.621385760 (576h32m1.492468805s) [noise=true , address=45.91.116.9:4264 , piece_id=2UORUTXJTNKR4ORJPUFH5LTRKN62ZYERE7R4OKM6U3CN2BMR7HGA, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.633642309 (576h32m1.504725354s) [noise=true , address=45.91.116.14:4191 , piece_id=EMSGCRYFMI7LOUOT3LFQP4EIXY2O4NDU3RTCVEQ3LGH2SXKMQYQA, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.633667165 (576h32m1.50475021s) [seg_pos={0 0}, upload=505421]: Segment reached optimal threshold of 13 pieces.
Dec 29 16:01:01.633970984 (576h32m1.505054039s) [noise=true , address=145.79.227.33:4871 , piece_id=USISPRAQJYRMKRCFG4I2SAUJRKW4WLV5DRXPY4TF3W2XIY6QJXNQ, seg_pos={0 0}, upload=505421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF
Dec 29 16:01:01.633986254 (576h32m1.505069299s) [noise=true , address=145.79.227.33:4871 , piece_id=USISPRAQJYRMKRCFG4I2SAUJRKW4WLV5DRXPY4TF3W2XIY6QJXNQ, seg_pos={0 0}, upload=505421]: Long tail cancellation due to optimal threshold
Dec 29 16:01:01.634007834 (576h32m1.505090879s) [noise=true , address=45.91.116.10:4083 , piece_id=IUPXAWEMC6NZ2BU62UUNXFJMLYAZDF46ZOPFQU2TCGN2UNQ7BUDA, seg_pos={0 0}, upload=505421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Dec 29 16:01:01.634015780 (576h32m1.505098825s) [noise=true , address=45.91.116.10:4083 , piece_id=IUPXAWEMC6NZ2BU62UUNXFJMLYAZDF46ZOPFQU2TCGN2UNQ7BUDA, seg_pos={0 0}, upload=505421]: Long tail cancellation due to optimal threshold
Dec 29 16:01:01.634212264 (576h32m1.505295309s) [noise=true , address=77.90.31.11:4554 , piece_id=TAABWVKM6E35CB332R4UZS47TWFULXLWJK67MPD6WZNXRTGJEHYA, seg_pos={0 0}, upload=505421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:33202->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33202->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33202->77.90.31.11:4554: use of closed network connection
Dec 29 16:01:01.634222032 (576h32m1.505305077s) [noise=true , address=77.90.31.11:4554 , piece_id=TAABWVKM6E35CB332R4UZS47TWFULXLWJK67MPD6WZNXRTGJEHYA, seg_pos={0 0}, upload=505421]: Long tail cancellation due to optimal threshold
Dec 29 16:01:01.634244004 (576h32m1.505327039s) [seg_pos={0 0}, upload=505421]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 16:01:01.634343674 (576h32m1.505426709s) [upload=505421]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 16:01:01.634350717 (576h32m1.505433782s) [upload=505421]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 16:01:01.927456770 (576h32m1.798539825s) [upload=505421]: Done uploading segments...
Dec 29 16:01:01.927486106 (576h32m1.798569151s) [upload=505421]: Upload finished. err: <nil>
Duration: 6.743910524s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...X..........................XX............................] duration:
43.774597391s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 16:02:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 43.774625606s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X................X........................................] duration:
18.768835798s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 16:02:45 UTC)
orbiter warning: context deadline exceeded
Duration: 18.768863721s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XXXXXXXXXXXXXX.XXXXXXXX] duration:
5.001042265s trace:
https://jaeger.dev.storj.io/trace/7efcb04fd1a7c0e
FAIL (2025-12-29 16:02:13 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001069287s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [..XXX..XX.X..X..XX.XX...X.X..XX....XX.......X....X.X......X.] duration:
5.950226688s trace:
https://jaeger.dev.storj.io/trace/713470deadd5c765
FAIL (2025-12-29 16:02:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: HSLAMA5PKLMEWPU43O432WDUJBX4ITOW52I4HFA573PAZASQIVPA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 2UORUTXJTNKR4ORJPUFH5LTRKN62ZYERE7R4OKM6U3CN2BMR7HGA) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: NMXHKF2WVPG7KF7OM54A5WUUZXT33V5VOUDAH4MV6FUJYLEITSPQ) context deadline exceeded
======LOGS======
Dec 29 16:00:59.061246935 (576h31m58.93232997s) [upload=505421]: Starting upload
Dec 29 16:00:59.061375279 (576h31m58.932458324s) [upload=505421]: Done starting upload
Dec 29 16:00:59.061477925 (576h31m58.93256097s) [upload=505421]: Uploading segments...
Dec 29 16:00:59.061577605 (576h31m58.93266065s) [upload=505421]: Got next segment. Inline: false
Dec 29 16:00:59.061611459 (576h31m58.932694494s) [upload=505421]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 16:00:59.061616889 (576h31m58.932699934s) [upload=505421]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 16:00:59.308224802 (576h31m59.179307837s) [seg_pos={0 0}, upload=505421]: Begin upload segment...
Dec 29 16:00:59.308278834 (576h31m59.179361879s) [seg_pos={0 0}, upload=505421]: Done begin upload segment.
Dec 29 16:00:59.308315033 (576h31m59.179398078s) [noise=true , address=145.79.227.16:4910 , piece_id=6TUL54UVPKW42PT24ANUSMZFCCN56LOHPKNPKNL32SKMWPGOMC3Q, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.308355600 (576h31m59.179438645s) [noise=true , address=145.79.227.18:4007 , piece_id=ZTEJZFB7LA7D22XCZSVSEWW7NMQGBVYVLWYDTMKRUIQSRNEBM7OA, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.308865201 (576h31m59.179948246s) [noise=true , address=145.79.227.2:4113 , piece_id=SB6LSZ4RVM2UR5JPAAF7R4CMU6PDJGULYJBQILALRTOIQDCUASRQ, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.308874338 (576h31m59.179957383s) [noise=true , address=77.90.31.21:4878 , piece_id=Y62OMBKCLT6ASX7YY6DIAGXCEMHHJ5TGJGQ5MAJ4JTXN3K5ETN4Q, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.309405410 (576h31m59.180488455s) [noise=true , address=45.91.116.9:4264 , piece_id=2UORUTXJTNKR4ORJPUFH5LTRKN62ZYERE7R4OKM6U3CN2BMR7HGA, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.309549134 (576h31m59.180632179s) [noise=true , address=77.90.31.13:4629 , piece_id=NMXHKF2WVPG7KF7OM54A5WUUZXT33V5VOUDAH4MV6FUJYLEITSPQ, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.309850056 (576h31m59.180933101s) [noise=true , address=145.79.227.32:4836 , piece_id=4WWQUDGPLIQSGMGIFMBDQWGTIU7Z6BAJ6MFX5BNPXRTFQ36WYTJQ, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.310054766 (576h31m59.181137811s) [noise=true , address=145.79.227.14:4837 , piece_id=SV7S4CG4QLIJTCZHET2JA2RX5TU6LGF6XZTEIIXG436ULYNZANSQ, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.310324079 (576h31m59.181407134s) [noise=true , address=145.79.227.38:4978 , piece_id=E7JGEIVTOFC6NB7WILMH3EHKFBMS6YIQFMSL57MOFQMVR67D2VKA, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.310621887 (576h31m59.181704932s) [noise=true , address=145.79.227.33:4871 , piece_id=USISPRAQJYRMKRCFG4I2SAUJRKW4WLV5DRXPY4TF3W2XIY6QJXNQ, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.310818741 (576h31m59.181901786s) [noise=true , address=45.91.116.14:4191 , piece_id=EMSGCRYFMI7LOUOT3LFQP4EIXY2O4NDU3RTCVEQ3LGH2SXKMQYQA, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.311091601 (576h31m59.182174646s) [noise=true , address=77.90.31.11:4554 , piece_id=TAABWVKM6E35CB332R4UZS47TWFULXLWJK67MPD6WZNXRTGJEHYA, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.311223231 (576h31m59.182306276s) [noise=true , address=145.79.227.28:4726 , piece_id=3FXO4JNTT3DYATMLLXYHYH33NESEMQ7FOY4SKK4WT37BLVNDSACQ, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.311573268 (576h31m59.182656313s) [noise=true , address=45.91.116.10:4083 , piece_id=IUPXAWEMC6NZ2BU62UUNXFJMLYAZDF46ZOPFQU2TCGN2UNQ7BUDA, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.311638462 (576h31m59.182721497s) [noise=true , address=145.79.227.6:4259 , piece_id=CZTZBSLSWG7ZGB4YOLCE67SDDW2J56FUATQ7AYKKXFHNSQG5H7EQ, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.312460958 (576h31m59.183544003s) [noise=true , address=45.91.116.12:4226 , piece_id=HSLAMA5PKLMEWPU43O432WDUJBX4ITOW52I4HFA573PAZASQIVPA, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:01:01.000597506 (576h32m0.871680572s) [upload=505421]: Next returned nil segment
Dec 29 16:01:01.000621773 (576h32m0.871704838s) [upload=505421]: Waiting for error group managing segments...
Dec 29 16:01:01.286374995 (576h32m1.15745804s) [noise=true , address=145.79.227.2:4113 , piece_id=SB6LSZ4RVM2UR5JPAAF7R4CMU6PDJGULYJBQILALRTOIQDCUASRQ, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.299456593 (576h32m1.170539639s) [noise=true , address=145.79.227.18:4007 , piece_id=ZTEJZFB7LA7D22XCZSVSEWW7NMQGBVYVLWYDTMKRUIQSRNEBM7OA, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.299739893 (576h32m1.170822928s) [noise=true , address=145.79.227.28:4726 , piece_id=3FXO4JNTT3DYATMLLXYHYH33NESEMQ7FOY4SKK4WT37BLVNDSACQ, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.301403240 (576h32m1.172486285s) [noise=true , address=145.79.227.32:4836 , piece_id=4WWQUDGPLIQSGMGIFMBDQWGTIU7Z6BAJ6MFX5BNPXRTFQ36WYTJQ, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.310564936 (576h32m1.181647981s) [noise=true , address=145.79.227.6:4259 , piece_id=CZTZBSLSWG7ZGB4YOLCE67SDDW2J56FUATQ7AYKKXFHNSQG5H7EQ, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.319403225 (576h32m1.19048627s) [noise=true , address=145.79.227.14:4837 , piece_id=SV7S4CG4QLIJTCZHET2JA2RX5TU6LGF6XZTEIIXG436ULYNZANSQ, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.321985914 (576h32m1.193068959s) [noise=true , address=145.79.227.16:4910 , piece_id=6TUL54UVPKW42PT24ANUSMZFCCN56LOHPKNPKNL32SKMWPGOMC3Q, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.324207393 (576h32m1.195290438s) [noise=true , address=145.79.227.38:4978 , piece_id=E7JGEIVTOFC6NB7WILMH3EHKFBMS6YIQFMSL57MOFQMVR67D2VKA, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.505165477 (576h32m1.376248522s) [noise=true , address=77.90.31.13:4629 , piece_id=NMXHKF2WVPG7KF7OM54A5WUUZXT33V5VOUDAH4MV6FUJYLEITSPQ, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.581245907 (576h32m1.452328952s) [noise=true , address=77.90.31.21:4878 , piece_id=Y62OMBKCLT6ASX7YY6DIAGXCEMHHJ5TGJGQ5MAJ4JTXN3K5ETN4Q, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.615014799 (576h32m1.486097844s) [noise=true , address=45.91.116.12:4226 , piece_id=HSLAMA5PKLMEWPU43O432WDUJBX4ITOW52I4HFA573PAZASQIVPA, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.621385760 (576h32m1.492468805s) [noise=true , address=45.91.116.9:4264 , piece_id=2UORUTXJTNKR4ORJPUFH5LTRKN62ZYERE7R4OKM6U3CN2BMR7HGA, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.633642309 (576h32m1.504725354s) [noise=true , address=45.91.116.14:4191 , piece_id=EMSGCRYFMI7LOUOT3LFQP4EIXY2O4NDU3RTCVEQ3LGH2SXKMQYQA, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.633667165 (576h32m1.50475021s) [seg_pos={0 0}, upload=505421]: Segment reached optimal threshold of 13 pieces.
Dec 29 16:01:01.633970984 (576h32m1.505054039s) [noise=true , address=145.79.227.33:4871 , piece_id=USISPRAQJYRMKRCFG4I2SAUJRKW4WLV5DRXPY4TF3W2XIY6QJXNQ, seg_pos={0 0}, upload=505421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF
Dec 29 16:01:01.633986254 (576h32m1.505069299s) [noise=true , address=145.79.227.33:4871 , piece_id=USISPRAQJYRMKRCFG4I2SAUJRKW4WLV5DRXPY4TF3W2XIY6QJXNQ, seg_pos={0 0}, upload=505421]: Long tail cancellation due to optimal threshold
Dec 29 16:01:01.634007834 (576h32m1.505090879s) [noise=true , address=45.91.116.10:4083 , piece_id=IUPXAWEMC6NZ2BU62UUNXFJMLYAZDF46ZOPFQU2TCGN2UNQ7BUDA, seg_pos={0 0}, upload=505421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Dec 29 16:01:01.634015780 (576h32m1.505098825s) [noise=true , address=45.91.116.10:4083 , piece_id=IUPXAWEMC6NZ2BU62UUNXFJMLYAZDF46ZOPFQU2TCGN2UNQ7BUDA, seg_pos={0 0}, upload=505421]: Long tail cancellation due to optimal threshold
Dec 29 16:01:01.634212264 (576h32m1.505295309s) [noise=true , address=77.90.31.11:4554 , piece_id=TAABWVKM6E35CB332R4UZS47TWFULXLWJK67MPD6WZNXRTGJEHYA, seg_pos={0 0}, upload=505421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:33202->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33202->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33202->77.90.31.11:4554: use of closed network connection
Dec 29 16:01:01.634222032 (576h32m1.505305077s) [noise=true , address=77.90.31.11:4554 , piece_id=TAABWVKM6E35CB332R4UZS47TWFULXLWJK67MPD6WZNXRTGJEHYA, seg_pos={0 0}, upload=505421]: Long tail cancellation due to optimal threshold
Dec 29 16:01:01.634244004 (576h32m1.505327039s) [seg_pos={0 0}, upload=505421]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 16:01:01.634343674 (576h32m1.505426709s) [upload=505421]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 16:01:01.634350717 (576h32m1.505433782s) [upload=505421]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 16:01:01.927456770 (576h32m1.798539825s) [upload=505421]: Done uploading segments...
Dec 29 16:01:01.927486106 (576h32m1.798569151s) [upload=505421]: Upload finished. err: <nil>
Duration: 5.950252137s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X................X.......................................] duration:
13.300462629s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 16:01:46 UTC)
orbiter warning: context deadline exceeded
Duration: 13.300490902s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
47 out of 60 is failed (threshold=6) [.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XXXXXXXXXXXXXX.XXXXXXX] duration:
5.001274811s trace:
https://jaeger.dev.storj.io/trace/7032d24cd0ffffe4
FAIL (2025-12-29 16:01:40 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001298075s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [...XXX..XX.X..X..XX.XX...X.X..XX....XX.......X....X.X......X] duration:
30.003208406s trace:
https://jaeger.dev.storj.io/trace/713470deadd5c765
FAIL (2025-12-29 16:01:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: HSLAMA5PKLMEWPU43O432WDUJBX4ITOW52I4HFA573PAZASQIVPA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 2UORUTXJTNKR4ORJPUFH5LTRKN62ZYERE7R4OKM6U3CN2BMR7HGA) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: NMXHKF2WVPG7KF7OM54A5WUUZXT33V5VOUDAH4MV6FUJYLEITSPQ) context deadline exceeded
======LOGS======
Dec 29 16:00:59.061246935 (576h31m58.93232997s) [upload=505421]: Starting upload
Dec 29 16:00:59.061375279 (576h31m58.932458324s) [upload=505421]: Done starting upload
Dec 29 16:00:59.061477925 (576h31m58.93256097s) [upload=505421]: Uploading segments...
Dec 29 16:00:59.061577605 (576h31m58.93266065s) [upload=505421]: Got next segment. Inline: false
Dec 29 16:00:59.061611459 (576h31m58.932694494s) [upload=505421]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 16:00:59.061616889 (576h31m58.932699934s) [upload=505421]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 16:00:59.308224802 (576h31m59.179307837s) [seg_pos={0 0}, upload=505421]: Begin upload segment...
Dec 29 16:00:59.308278834 (576h31m59.179361879s) [seg_pos={0 0}, upload=505421]: Done begin upload segment.
Dec 29 16:00:59.308315033 (576h31m59.179398078s) [noise=true , address=145.79.227.16:4910 , piece_id=6TUL54UVPKW42PT24ANUSMZFCCN56LOHPKNPKNL32SKMWPGOMC3Q, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.308355600 (576h31m59.179438645s) [noise=true , address=145.79.227.18:4007 , piece_id=ZTEJZFB7LA7D22XCZSVSEWW7NMQGBVYVLWYDTMKRUIQSRNEBM7OA, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.308865201 (576h31m59.179948246s) [noise=true , address=145.79.227.2:4113 , piece_id=SB6LSZ4RVM2UR5JPAAF7R4CMU6PDJGULYJBQILALRTOIQDCUASRQ, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.308874338 (576h31m59.179957383s) [noise=true , address=77.90.31.21:4878 , piece_id=Y62OMBKCLT6ASX7YY6DIAGXCEMHHJ5TGJGQ5MAJ4JTXN3K5ETN4Q, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.309405410 (576h31m59.180488455s) [noise=true , address=45.91.116.9:4264 , piece_id=2UORUTXJTNKR4ORJPUFH5LTRKN62ZYERE7R4OKM6U3CN2BMR7HGA, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.309549134 (576h31m59.180632179s) [noise=true , address=77.90.31.13:4629 , piece_id=NMXHKF2WVPG7KF7OM54A5WUUZXT33V5VOUDAH4MV6FUJYLEITSPQ, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.309850056 (576h31m59.180933101s) [noise=true , address=145.79.227.32:4836 , piece_id=4WWQUDGPLIQSGMGIFMBDQWGTIU7Z6BAJ6MFX5BNPXRTFQ36WYTJQ, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.310054766 (576h31m59.181137811s) [noise=true , address=145.79.227.14:4837 , piece_id=SV7S4CG4QLIJTCZHET2JA2RX5TU6LGF6XZTEIIXG436ULYNZANSQ, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.310324079 (576h31m59.181407134s) [noise=true , address=145.79.227.38:4978 , piece_id=E7JGEIVTOFC6NB7WILMH3EHKFBMS6YIQFMSL57MOFQMVR67D2VKA, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.310621887 (576h31m59.181704932s) [noise=true , address=145.79.227.33:4871 , piece_id=USISPRAQJYRMKRCFG4I2SAUJRKW4WLV5DRXPY4TF3W2XIY6QJXNQ, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.310818741 (576h31m59.181901786s) [noise=true , address=45.91.116.14:4191 , piece_id=EMSGCRYFMI7LOUOT3LFQP4EIXY2O4NDU3RTCVEQ3LGH2SXKMQYQA, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.311091601 (576h31m59.182174646s) [noise=true , address=77.90.31.11:4554 , piece_id=TAABWVKM6E35CB332R4UZS47TWFULXLWJK67MPD6WZNXRTGJEHYA, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.311223231 (576h31m59.182306276s) [noise=true , address=145.79.227.28:4726 , piece_id=3FXO4JNTT3DYATMLLXYHYH33NESEMQ7FOY4SKK4WT37BLVNDSACQ, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.311573268 (576h31m59.182656313s) [noise=true , address=45.91.116.10:4083 , piece_id=IUPXAWEMC6NZ2BU62UUNXFJMLYAZDF46ZOPFQU2TCGN2UNQ7BUDA, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.311638462 (576h31m59.182721497s) [noise=true , address=145.79.227.6:4259 , piece_id=CZTZBSLSWG7ZGB4YOLCE67SDDW2J56FUATQ7AYKKXFHNSQG5H7EQ, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:00:59.312460958 (576h31m59.183544003s) [noise=true , address=45.91.116.12:4226 , piece_id=HSLAMA5PKLMEWPU43O432WDUJBX4ITOW52I4HFA573PAZASQIVPA, seg_pos={0 0}, upload=505421]: Uploading piece...
Dec 29 16:01:01.000597506 (576h32m0.871680572s) [upload=505421]: Next returned nil segment
Dec 29 16:01:01.000621773 (576h32m0.871704838s) [upload=505421]: Waiting for error group managing segments...
Dec 29 16:01:01.286374995 (576h32m1.15745804s) [noise=true , address=145.79.227.2:4113 , piece_id=SB6LSZ4RVM2UR5JPAAF7R4CMU6PDJGULYJBQILALRTOIQDCUASRQ, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.299456593 (576h32m1.170539639s) [noise=true , address=145.79.227.18:4007 , piece_id=ZTEJZFB7LA7D22XCZSVSEWW7NMQGBVYVLWYDTMKRUIQSRNEBM7OA, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.299739893 (576h32m1.170822928s) [noise=true , address=145.79.227.28:4726 , piece_id=3FXO4JNTT3DYATMLLXYHYH33NESEMQ7FOY4SKK4WT37BLVNDSACQ, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.301403240 (576h32m1.172486285s) [noise=true , address=145.79.227.32:4836 , piece_id=4WWQUDGPLIQSGMGIFMBDQWGTIU7Z6BAJ6MFX5BNPXRTFQ36WYTJQ, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.310564936 (576h32m1.181647981s) [noise=true , address=145.79.227.6:4259 , piece_id=CZTZBSLSWG7ZGB4YOLCE67SDDW2J56FUATQ7AYKKXFHNSQG5H7EQ, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.319403225 (576h32m1.19048627s) [noise=true , address=145.79.227.14:4837 , piece_id=SV7S4CG4QLIJTCZHET2JA2RX5TU6LGF6XZTEIIXG436ULYNZANSQ, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.321985914 (576h32m1.193068959s) [noise=true , address=145.79.227.16:4910 , piece_id=6TUL54UVPKW42PT24ANUSMZFCCN56LOHPKNPKNL32SKMWPGOMC3Q, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.324207393 (576h32m1.195290438s) [noise=true , address=145.79.227.38:4978 , piece_id=E7JGEIVTOFC6NB7WILMH3EHKFBMS6YIQFMSL57MOFQMVR67D2VKA, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.505165477 (576h32m1.376248522s) [noise=true , address=77.90.31.13:4629 , piece_id=NMXHKF2WVPG7KF7OM54A5WUUZXT33V5VOUDAH4MV6FUJYLEITSPQ, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.581245907 (576h32m1.452328952s) [noise=true , address=77.90.31.21:4878 , piece_id=Y62OMBKCLT6ASX7YY6DIAGXCEMHHJ5TGJGQ5MAJ4JTXN3K5ETN4Q, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.615014799 (576h32m1.486097844s) [noise=true , address=45.91.116.12:4226 , piece_id=HSLAMA5PKLMEWPU43O432WDUJBX4ITOW52I4HFA573PAZASQIVPA, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.621385760 (576h32m1.492468805s) [noise=true , address=45.91.116.9:4264 , piece_id=2UORUTXJTNKR4ORJPUFH5LTRKN62ZYERE7R4OKM6U3CN2BMR7HGA, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.633642309 (576h32m1.504725354s) [noise=true , address=45.91.116.14:4191 , piece_id=EMSGCRYFMI7LOUOT3LFQP4EIXY2O4NDU3RTCVEQ3LGH2SXKMQYQA, seg_pos={0 0}, upload=505421]: Done uploading piece. err: <nil>
Dec 29 16:01:01.633667165 (576h32m1.50475021s) [seg_pos={0 0}, upload=505421]: Segment reached optimal threshold of 13 pieces.
Dec 29 16:01:01.633970984 (576h32m1.505054039s) [noise=true , address=145.79.227.33:4871 , piece_id=USISPRAQJYRMKRCFG4I2SAUJRKW4WLV5DRXPY4TF3W2XIY6QJXNQ, seg_pos={0 0}, upload=505421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF
Dec 29 16:01:01.633986254 (576h32m1.505069299s) [noise=true , address=145.79.227.33:4871 , piece_id=USISPRAQJYRMKRCFG4I2SAUJRKW4WLV5DRXPY4TF3W2XIY6QJXNQ, seg_pos={0 0}, upload=505421]: Long tail cancellation due to optimal threshold
Dec 29 16:01:01.634007834 (576h32m1.505090879s) [noise=true , address=45.91.116.10:4083 , piece_id=IUPXAWEMC6NZ2BU62UUNXFJMLYAZDF46ZOPFQU2TCGN2UNQ7BUDA, seg_pos={0 0}, upload=505421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Dec 29 16:01:01.634015780 (576h32m1.505098825s) [noise=true , address=45.91.116.10:4083 , piece_id=IUPXAWEMC6NZ2BU62UUNXFJMLYAZDF46ZOPFQU2TCGN2UNQ7BUDA, seg_pos={0 0}, upload=505421]: Long tail cancellation due to optimal threshold
Dec 29 16:01:01.634212264 (576h32m1.505295309s) [noise=true , address=77.90.31.11:4554 , piece_id=TAABWVKM6E35CB332R4UZS47TWFULXLWJK67MPD6WZNXRTGJEHYA, seg_pos={0 0}, upload=505421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:33202->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33202->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33202->77.90.31.11:4554: use of closed network connection
Dec 29 16:01:01.634222032 (576h32m1.505305077s) [noise=true , address=77.90.31.11:4554 , piece_id=TAABWVKM6E35CB332R4UZS47TWFULXLWJK67MPD6WZNXRTGJEHYA, seg_pos={0 0}, upload=505421]: Long tail cancellation due to optimal threshold
Dec 29 16:01:01.634244004 (576h32m1.505327039s) [seg_pos={0 0}, upload=505421]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 16:01:01.634343674 (576h32m1.505426709s) [upload=505421]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 16:01:01.634350717 (576h32m1.505433782s) [upload=505421]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 16:01:01.927456770 (576h32m1.798539825s) [upload=505421]: Done uploading segments...
Dec 29 16:01:01.927486106 (576h32m1.798569151s) [upload=505421]: Upload finished. err: <nil>
Duration: 30.003232072s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [....X..........................XX...........................] duration:
33.030628798s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 16:01:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 33.030645431s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....X..........................XX..........................] duration:
41.336785028s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 16:00:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 41.336802491s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....X................X......................................] duration:
16.461091787s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 16:00:49 UTC)
orbiter warning: context deadline exceeded
Duration: 16.461194563s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
47 out of 60 is failed (threshold=6) [X.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XXXXXXXXXXXXXX.XXXXXX] duration:
5.001247568s trace:
https://jaeger.dev.storj.io/trace/3f1f54231f2c88c3
FAIL (2025-12-29 16:00:40 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001265823s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [.X.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XXXXXXXXXXXXXX.XXXXX] duration:
5.001077734s trace:
https://jaeger.dev.storj.io/trace/1d2bcdc818090e45
FAIL (2025-12-29 16:00:20 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001107881s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [......X..........................XX.........................] duration:
33.847873601s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 16:00:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 33.847890885s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....X................X.....................................] duration:
13.053603117s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 16:00:09 UTC)
orbiter warning: context deadline exceeded
Duration: 13.053628566s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [X...XXX..XX.X..X..XX.XX...X.X..XX....XX.......X....X.X......] duration:
6.02027276s trace:
https://jaeger.dev.storj.io/trace/5a70c6bfa63ee1d5
FAIL (2025-12-29 16:00:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: LMKURYTYLS7ZLFUDA7AY3BYPA234AEIHDMBVKRN5OPTD52CES6BQ) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: 6HU7UEX2USEBWQ3RWPPKENRGL2TLC7BCT6UTXVRZEZQUTQBKHGLA) context deadline exceeded
======LOGS======
ec 29 15:56:33.292748031 (576h27m33.163831086s) [upload=505352]: Starting upload
Dec 29 15:56:33.292802995 (576h27m33.16388604s) [upload=505352]: Done starting upload
Dec 29 15:56:33.292870354 (576h27m33.163953399s) [upload=505352]: Uploading segments...
Dec 29 15:56:33.293052390 (576h27m33.164135425s) [upload=505352]: Got next segment. Inline: false
Dec 29 15:56:33.293081686 (576h27m33.164164721s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:56:33.293086184 (576h27m33.164169219s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:56:33.543074633 (576h27m33.414157678s) [seg_pos={0 0}, upload=505352]: Begin upload segment...
Dec 29 15:56:33.543129268 (576h27m33.414212303s) [seg_pos={0 0}, upload=505352]: Done begin upload segment.
Dec 29 15:56:33.543166018 (576h27m33.414249063s) [noise=true , address=145.79.227.5:4222 , piece_id=NOHTETEZOWG2FHCIUBH6T6FTCENFUICJZ7HKRSSP4SOKJDTXA5FQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.543223748 (576h27m33.414306793s) [noise=true , address=45.91.116.8:4045 , piece_id=JCUDRBXV4SHGLFUMYCPAZXEN4C4T3HP6VXOMAO4Y3XJTLQX6IZVA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.543852325 (576h27m33.41493537s) [noise=true , address=77.90.31.18:4805 , piece_id=6HU7UEX2USEBWQ3RWPPKENRGL2TLC7BCT6UTXVRZEZQUTQBKHGLA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.544288615 (576h27m33.41537166s) [noise=true , address=77.90.31.14:4662 , piece_id=B67ICRUA7HPJFKS2DNDR4FQA2XHTH7CLZ7CPGNEAJULWW62W7JUA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.544691412 (576h27m33.415774447s) [noise=true , address=145.79.227.42:4121 , piece_id=SDH6PZRLR6HKXXQZ7QISEYZAZBPEP4HFBSAAZY23XVGLXJREJBMA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545158792 (576h27m33.416241837s) [noise=true , address=45.91.116.13:4156 , piece_id=WZGYZDPHV5EXGONPSF5FS57DOSO4CGTBB3GOLK5G3IN5NLRZ3NZQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545530620 (576h27m33.416613675s) [noise=true , address=145.79.227.11:4730 , piece_id=OOZG5W4UTT4DFS2OKAFV2KD3KSV57GSSN5RPD6COULXEA7KIA45Q, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545966289 (576h27m33.417049334s) [noise=true , address=45.91.116.12:4225 , piece_id=67TYC4MDAXRXFAJZC6TAJLZNJPC54LBN545BTYTWBH5OJT7DH35A, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545981719 (576h27m33.417064764s) [noise=true , address=145.79.227.52:4447 , piece_id=DCJSQWH5AUQQ466JPFVUZQJU2EISTTUB56XN3RY3QMJ3GJSYYT5Q, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.546368385 (576h27m33.417451421s) [noise=true , address=45.91.116.7:4011 , piece_id=SYKZRWQP7TXEXDXJVQAWZIEIX3B6AMT27XSHCFLEYIB2AA2T4TVQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.546711990 (576h27m33.417795035s) [noise=true , address=77.90.31.15:4697 , piece_id=7ZGRV7LGB6TR6UIJXDPCPBHXIGAA7RF3VWXQ5AOYRVUCHWNRDLGQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.546993867 (576h27m33.418076912s) [noise=true , address=45.91.116.5:4298 , piece_id=LMKURYTYLS7ZLFUDA7AY3BYPA234AEIHDMBVKRN5OPTD52CES6BQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.547300671 (576h27m33.418383706s) [noise=true , address=145.79.227.3:4150 , piece_id=457PQSGEFT6UKHV44ZFXEPRREYQOV6ILIWJWJCTQCBRXF7EIXXAA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.547598437 (576h27m33.418681472s) [noise=true , address=45.91.116.9:4261 , piece_id=RPNFXJQOUCXWVDOBLEOKYLEYV73QZYYYVTO7V5N3HPKIUQ2ZPN3Q, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.548170878 (576h27m33.419253913s) [noise=true , address=145.79.227.6:4259 , piece_id=BPYLWD3AMOC35GJPPC6E2VCKXIJX3OEEFJONLUYQRGFBES5NSNSA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.548496297 (576h27m33.419579332s) [noise=true , address=145.79.227.15:4873 , piece_id=E3DWN2MNIB4QQYUIFVWZLAHTI3OLQPKYRUREJOCUJLD555TL77NA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:35.244735136 (576h27m35.115818181s) [upload=505352]: Next returned nil segment
Dec 29 15:56:35.244750024 (576h27m35.115833079s) [upload=505352]: Waiting for error group managing segments...
Dec 29 15:56:35.528064171 (576h27m35.399147216s) [noise=true , address=145.79.227.5:4222 , piece_id=NOHTETEZOWG2FHCIUBH6T6FTCENFUICJZ7HKRSSP4SOKJDTXA5FQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.529417919 (576h27m35.400500964s) [noise=true , address=145.79.227.11:4730 , piece_id=OOZG5W4UTT4DFS2OKAFV2KD3KSV57GSSN5RPD6COULXEA7KIA45Q, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.540921563 (576h27m35.412004608s) [noise=true , address=145.79.227.6:4259 , piece_id=BPYLWD3AMOC35GJPPC6E2VCKXIJX3OEEFJONLUYQRGFBES5NSNSA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.546841225 (576h27m35.41792426s) [noise=true , address=145.79.227.15:4873 , piece_id=E3DWN2MNIB4QQYUIFVWZLAHTI3OLQPKYRUREJOCUJLD555TL77NA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.550607858 (576h27m35.421690903s) [noise=true , address=145.79.227.42:4121 , piece_id=SDH6PZRLR6HKXXQZ7QISEYZAZBPEP4HFBSAAZY23XVGLXJREJBMA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.562935632 (576h27m35.434018677s) [noise=true , address=145.79.227.52:4447 , piece_id=DCJSQWH5AUQQ466JPFVUZQJU2EISTTUB56XN3RY3QMJ3GJSYYT5Q, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.767765121 (576h27m35.638848166s) [noise=true , address=45.91.116.7:4011 , piece_id=SYKZRWQP7TXEXDXJVQAWZIEIX3B6AMT27XSHCFLEYIB2AA2T4TVQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.787755876 (576h27m35.658838911s) [noise=true , address=77.90.31.15:4697 , piece_id=7ZGRV7LGB6TR6UIJXDPCPBHXIGAA7RF3VWXQ5AOYRVUCHWNRDLGQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.807924060 (576h27m35.679007095s) [noise=true , address=77.90.31.18:4805 , piece_id=6HU7UEX2USEBWQ3RWPPKENRGL2TLC7BCT6UTXVRZEZQUTQBKHGLA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.829697982 (576h27m35.700781037s) [noise=true , address=77.90.31.14:4662 , piece_id=B67ICRUA7HPJFKS2DNDR4FQA2XHTH7CLZ7CPGNEAJULWW62W7JUA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.830640226 (576h27m35.701723271s) [noise=true , address=45.91.116.5:4298 , piece_id=LMKURYTYLS7ZLFUDA7AY3BYPA234AEIHDMBVKRN5OPTD52CES6BQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.910192497 (576h27m35.781275542s) [noise=true , address=45.91.116.12:4225 , piece_id=67TYC4MDAXRXFAJZC6TAJLZNJPC54LBN545BTYTWBH5OJT7DH35A, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:36.504292439 (576h27m36.375375484s) [noise=true , address=145.79.227.3:4150 , piece_id=457PQSGEFT6UKHV44ZFXEPRREYQOV6ILIWJWJCTQCBRXF7EIXXAA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:36.504319370 (576h27m36.375402416s) [seg_pos={0 0}, upload=505352]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:56:36.504544049 (576h27m36.375627094s) [noise=true , address=45.91.116.13:4156 , piece_id=WZGYZDPHV5EXGONPSF5FS57DOSO4CGTBB3GOLK5G3IN5NLRZ3NZQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 29 15:56:36.504552885 (576h27m36.37563592s) [noise=true , address=45.91.116.13:4156 , piece_id=WZGYZDPHV5EXGONPSF5FS57DOSO4CGTBB3GOLK5G3IN5NLRZ3NZQ, seg_pos={0 0}, upload=505352]: Long tail cancellation due to optimal threshold
Dec 29 15:56:36.504837718 (576h27m36.375920763s) [noise=true , address=45.91.116.8:4045 , piece_id=JCUDRBXV4SHGLFUMYCPAZXEN4C4T3HP6VXOMAO4Y3XJTLQX6IZVA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 29 15:56:36.504881811 (576h27m36.375964846s) [noise=true , address=45.91.116.8:4045 , piece_id=JCUDRBXV4SHGLFUMYCPAZXEN4C4T3HP6VXOMAO4Y3XJTLQX6IZVA, seg_pos={0 0}, upload=505352]: Long tail cancellation due to optimal threshold
Dec 29 15:56:36.504916918 (576h27m36.375999953s) [noise=true , address=45.91.116.9:4261 , piece_id=RPNFXJQOUCXWVDOBLEOKYLEYV73QZYYYVTO7V5N3HPKIUQ2ZPN3Q, seg_pos={0 0}, upload=505352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:47230->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:47230->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->45.91.116.9:4261: use of closed network connection
Dec 29 15:56:36.504927569 (576h27m36.376010614s) [noise=true , address=45.91.116.9:4261 , piece_id=RPNFXJQOUCXWVDOBLEOKYLEYV73QZYYYVTO7V5N3HPKIUQ2ZPN3Q, seg_pos={0 0}, upload=505352]: Long tail cancellation due to optimal threshold
Dec 29 15:56:36.505040294 (576h27m36.376123339s) [seg_pos={0 0}, upload=505352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:56:36.505185149 (576h27m36.376268184s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:56:36.505211309 (576h27m36.376294344s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:56:36.807637398 (576h27m36.678720473s) [upload=505352]: Done uploading segments...
Dec 29 15:56:36.807697783 (576h27m36.678780828s) [upload=505352]: Upload finished. err: <nil>
Duration: 6.020322575s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......X..........................XX........................] duration:
47.019712046s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 15:59:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 47.019735559s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X................X....................................] duration:
16.112357269s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 15:59:23 UTC)
orbiter warning: context deadline exceeded
Duration: 16.112378429s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.X...XXX..XX.X..X..XX.XX...X.X..XX....XX.......X....X.X.....] duration:
5.876252512s trace:
https://jaeger.dev.storj.io/trace/5a70c6bfa63ee1d5
FAIL (2025-12-29 15:59:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: LMKURYTYLS7ZLFUDA7AY3BYPA234AEIHDMBVKRN5OPTD52CES6BQ) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: 6HU7UEX2USEBWQ3RWPPKENRGL2TLC7BCT6UTXVRZEZQUTQBKHGLA) context deadline exceeded
======LOGS======
ec 29 15:56:33.292748031 (576h27m33.163831086s) [upload=505352]: Starting upload
Dec 29 15:56:33.292802995 (576h27m33.16388604s) [upload=505352]: Done starting upload
Dec 29 15:56:33.292870354 (576h27m33.163953399s) [upload=505352]: Uploading segments...
Dec 29 15:56:33.293052390 (576h27m33.164135425s) [upload=505352]: Got next segment. Inline: false
Dec 29 15:56:33.293081686 (576h27m33.164164721s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:56:33.293086184 (576h27m33.164169219s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:56:33.543074633 (576h27m33.414157678s) [seg_pos={0 0}, upload=505352]: Begin upload segment...
Dec 29 15:56:33.543129268 (576h27m33.414212303s) [seg_pos={0 0}, upload=505352]: Done begin upload segment.
Dec 29 15:56:33.543166018 (576h27m33.414249063s) [noise=true , address=145.79.227.5:4222 , piece_id=NOHTETEZOWG2FHCIUBH6T6FTCENFUICJZ7HKRSSP4SOKJDTXA5FQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.543223748 (576h27m33.414306793s) [noise=true , address=45.91.116.8:4045 , piece_id=JCUDRBXV4SHGLFUMYCPAZXEN4C4T3HP6VXOMAO4Y3XJTLQX6IZVA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.543852325 (576h27m33.41493537s) [noise=true , address=77.90.31.18:4805 , piece_id=6HU7UEX2USEBWQ3RWPPKENRGL2TLC7BCT6UTXVRZEZQUTQBKHGLA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.544288615 (576h27m33.41537166s) [noise=true , address=77.90.31.14:4662 , piece_id=B67ICRUA7HPJFKS2DNDR4FQA2XHTH7CLZ7CPGNEAJULWW62W7JUA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.544691412 (576h27m33.415774447s) [noise=true , address=145.79.227.42:4121 , piece_id=SDH6PZRLR6HKXXQZ7QISEYZAZBPEP4HFBSAAZY23XVGLXJREJBMA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545158792 (576h27m33.416241837s) [noise=true , address=45.91.116.13:4156 , piece_id=WZGYZDPHV5EXGONPSF5FS57DOSO4CGTBB3GOLK5G3IN5NLRZ3NZQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545530620 (576h27m33.416613675s) [noise=true , address=145.79.227.11:4730 , piece_id=OOZG5W4UTT4DFS2OKAFV2KD3KSV57GSSN5RPD6COULXEA7KIA45Q, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545966289 (576h27m33.417049334s) [noise=true , address=45.91.116.12:4225 , piece_id=67TYC4MDAXRXFAJZC6TAJLZNJPC54LBN545BTYTWBH5OJT7DH35A, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545981719 (576h27m33.417064764s) [noise=true , address=145.79.227.52:4447 , piece_id=DCJSQWH5AUQQ466JPFVUZQJU2EISTTUB56XN3RY3QMJ3GJSYYT5Q, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.546368385 (576h27m33.417451421s) [noise=true , address=45.91.116.7:4011 , piece_id=SYKZRWQP7TXEXDXJVQAWZIEIX3B6AMT27XSHCFLEYIB2AA2T4TVQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.546711990 (576h27m33.417795035s) [noise=true , address=77.90.31.15:4697 , piece_id=7ZGRV7LGB6TR6UIJXDPCPBHXIGAA7RF3VWXQ5AOYRVUCHWNRDLGQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.546993867 (576h27m33.418076912s) [noise=true , address=45.91.116.5:4298 , piece_id=LMKURYTYLS7ZLFUDA7AY3BYPA234AEIHDMBVKRN5OPTD52CES6BQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.547300671 (576h27m33.418383706s) [noise=true , address=145.79.227.3:4150 , piece_id=457PQSGEFT6UKHV44ZFXEPRREYQOV6ILIWJWJCTQCBRXF7EIXXAA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.547598437 (576h27m33.418681472s) [noise=true , address=45.91.116.9:4261 , piece_id=RPNFXJQOUCXWVDOBLEOKYLEYV73QZYYYVTO7V5N3HPKIUQ2ZPN3Q, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.548170878 (576h27m33.419253913s) [noise=true , address=145.79.227.6:4259 , piece_id=BPYLWD3AMOC35GJPPC6E2VCKXIJX3OEEFJONLUYQRGFBES5NSNSA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.548496297 (576h27m33.419579332s) [noise=true , address=145.79.227.15:4873 , piece_id=E3DWN2MNIB4QQYUIFVWZLAHTI3OLQPKYRUREJOCUJLD555TL77NA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:35.244735136 (576h27m35.115818181s) [upload=505352]: Next returned nil segment
Dec 29 15:56:35.244750024 (576h27m35.115833079s) [upload=505352]: Waiting for error group managing segments...
Dec 29 15:56:35.528064171 (576h27m35.399147216s) [noise=true , address=145.79.227.5:4222 , piece_id=NOHTETEZOWG2FHCIUBH6T6FTCENFUICJZ7HKRSSP4SOKJDTXA5FQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.529417919 (576h27m35.400500964s) [noise=true , address=145.79.227.11:4730 , piece_id=OOZG5W4UTT4DFS2OKAFV2KD3KSV57GSSN5RPD6COULXEA7KIA45Q, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.540921563 (576h27m35.412004608s) [noise=true , address=145.79.227.6:4259 , piece_id=BPYLWD3AMOC35GJPPC6E2VCKXIJX3OEEFJONLUYQRGFBES5NSNSA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.546841225 (576h27m35.41792426s) [noise=true , address=145.79.227.15:4873 , piece_id=E3DWN2MNIB4QQYUIFVWZLAHTI3OLQPKYRUREJOCUJLD555TL77NA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.550607858 (576h27m35.421690903s) [noise=true , address=145.79.227.42:4121 , piece_id=SDH6PZRLR6HKXXQZ7QISEYZAZBPEP4HFBSAAZY23XVGLXJREJBMA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.562935632 (576h27m35.434018677s) [noise=true , address=145.79.227.52:4447 , piece_id=DCJSQWH5AUQQ466JPFVUZQJU2EISTTUB56XN3RY3QMJ3GJSYYT5Q, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.767765121 (576h27m35.638848166s) [noise=true , address=45.91.116.7:4011 , piece_id=SYKZRWQP7TXEXDXJVQAWZIEIX3B6AMT27XSHCFLEYIB2AA2T4TVQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.787755876 (576h27m35.658838911s) [noise=true , address=77.90.31.15:4697 , piece_id=7ZGRV7LGB6TR6UIJXDPCPBHXIGAA7RF3VWXQ5AOYRVUCHWNRDLGQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.807924060 (576h27m35.679007095s) [noise=true , address=77.90.31.18:4805 , piece_id=6HU7UEX2USEBWQ3RWPPKENRGL2TLC7BCT6UTXVRZEZQUTQBKHGLA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.829697982 (576h27m35.700781037s) [noise=true , address=77.90.31.14:4662 , piece_id=B67ICRUA7HPJFKS2DNDR4FQA2XHTH7CLZ7CPGNEAJULWW62W7JUA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.830640226 (576h27m35.701723271s) [noise=true , address=45.91.116.5:4298 , piece_id=LMKURYTYLS7ZLFUDA7AY3BYPA234AEIHDMBVKRN5OPTD52CES6BQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.910192497 (576h27m35.781275542s) [noise=true , address=45.91.116.12:4225 , piece_id=67TYC4MDAXRXFAJZC6TAJLZNJPC54LBN545BTYTWBH5OJT7DH35A, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:36.504292439 (576h27m36.375375484s) [noise=true , address=145.79.227.3:4150 , piece_id=457PQSGEFT6UKHV44ZFXEPRREYQOV6ILIWJWJCTQCBRXF7EIXXAA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:36.504319370 (576h27m36.375402416s) [seg_pos={0 0}, upload=505352]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:56:36.504544049 (576h27m36.375627094s) [noise=true , address=45.91.116.13:4156 , piece_id=WZGYZDPHV5EXGONPSF5FS57DOSO4CGTBB3GOLK5G3IN5NLRZ3NZQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 29 15:56:36.504552885 (576h27m36.37563592s) [noise=true , address=45.91.116.13:4156 , piece_id=WZGYZDPHV5EXGONPSF5FS57DOSO4CGTBB3GOLK5G3IN5NLRZ3NZQ, seg_pos={0 0}, upload=505352]: Long tail cancellation due to optimal threshold
Dec 29 15:56:36.504837718 (576h27m36.375920763s) [noise=true , address=45.91.116.8:4045 , piece_id=JCUDRBXV4SHGLFUMYCPAZXEN4C4T3HP6VXOMAO4Y3XJTLQX6IZVA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 29 15:56:36.504881811 (576h27m36.375964846s) [noise=true , address=45.91.116.8:4045 , piece_id=JCUDRBXV4SHGLFUMYCPAZXEN4C4T3HP6VXOMAO4Y3XJTLQX6IZVA, seg_pos={0 0}, upload=505352]: Long tail cancellation due to optimal threshold
Dec 29 15:56:36.504916918 (576h27m36.375999953s) [noise=true , address=45.91.116.9:4261 , piece_id=RPNFXJQOUCXWVDOBLEOKYLEYV73QZYYYVTO7V5N3HPKIUQ2ZPN3Q, seg_pos={0 0}, upload=505352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:47230->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:47230->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->45.91.116.9:4261: use of closed network connection
Dec 29 15:56:36.504927569 (576h27m36.376010614s) [noise=true , address=45.91.116.9:4261 , piece_id=RPNFXJQOUCXWVDOBLEOKYLEYV73QZYYYVTO7V5N3HPKIUQ2ZPN3Q, seg_pos={0 0}, upload=505352]: Long tail cancellation due to optimal threshold
Dec 29 15:56:36.505040294 (576h27m36.376123339s) [seg_pos={0 0}, upload=505352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:56:36.505185149 (576h27m36.376268184s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:56:36.505211309 (576h27m36.376294344s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:56:36.807637398 (576h27m36.678720473s) [upload=505352]: Done uploading segments...
Dec 29 15:56:36.807697783 (576h27m36.678780828s) [upload=505352]: Upload finished. err: <nil>
Duration: 5.87628285s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [..X.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XXXXXXXXXXXXXX.XXXX] duration:
5.001027528s trace:
https://jaeger.dev.storj.io/trace/7907dd6e0eecff9d
FAIL (2025-12-29 15:59:08 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001051894s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [..X...XXX..XX.X..X..XX.XX...X.X..XX....XX.......X....X.X....] duration:
5.760662224s trace:
https://jaeger.dev.storj.io/trace/5a70c6bfa63ee1d5
FAIL (2025-12-29 15:58:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: LMKURYTYLS7ZLFUDA7AY3BYPA234AEIHDMBVKRN5OPTD52CES6BQ) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: 6HU7UEX2USEBWQ3RWPPKENRGL2TLC7BCT6UTXVRZEZQUTQBKHGLA) context deadline exceeded
======LOGS======
ec 29 15:56:33.292748031 (576h27m33.163831086s) [upload=505352]: Starting upload
Dec 29 15:56:33.292802995 (576h27m33.16388604s) [upload=505352]: Done starting upload
Dec 29 15:56:33.292870354 (576h27m33.163953399s) [upload=505352]: Uploading segments...
Dec 29 15:56:33.293052390 (576h27m33.164135425s) [upload=505352]: Got next segment. Inline: false
Dec 29 15:56:33.293081686 (576h27m33.164164721s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:56:33.293086184 (576h27m33.164169219s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:56:33.543074633 (576h27m33.414157678s) [seg_pos={0 0}, upload=505352]: Begin upload segment...
Dec 29 15:56:33.543129268 (576h27m33.414212303s) [seg_pos={0 0}, upload=505352]: Done begin upload segment.
Dec 29 15:56:33.543166018 (576h27m33.414249063s) [noise=true , address=145.79.227.5:4222 , piece_id=NOHTETEZOWG2FHCIUBH6T6FTCENFUICJZ7HKRSSP4SOKJDTXA5FQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.543223748 (576h27m33.414306793s) [noise=true , address=45.91.116.8:4045 , piece_id=JCUDRBXV4SHGLFUMYCPAZXEN4C4T3HP6VXOMAO4Y3XJTLQX6IZVA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.543852325 (576h27m33.41493537s) [noise=true , address=77.90.31.18:4805 , piece_id=6HU7UEX2USEBWQ3RWPPKENRGL2TLC7BCT6UTXVRZEZQUTQBKHGLA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.544288615 (576h27m33.41537166s) [noise=true , address=77.90.31.14:4662 , piece_id=B67ICRUA7HPJFKS2DNDR4FQA2XHTH7CLZ7CPGNEAJULWW62W7JUA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.544691412 (576h27m33.415774447s) [noise=true , address=145.79.227.42:4121 , piece_id=SDH6PZRLR6HKXXQZ7QISEYZAZBPEP4HFBSAAZY23XVGLXJREJBMA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545158792 (576h27m33.416241837s) [noise=true , address=45.91.116.13:4156 , piece_id=WZGYZDPHV5EXGONPSF5FS57DOSO4CGTBB3GOLK5G3IN5NLRZ3NZQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545530620 (576h27m33.416613675s) [noise=true , address=145.79.227.11:4730 , piece_id=OOZG5W4UTT4DFS2OKAFV2KD3KSV57GSSN5RPD6COULXEA7KIA45Q, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545966289 (576h27m33.417049334s) [noise=true , address=45.91.116.12:4225 , piece_id=67TYC4MDAXRXFAJZC6TAJLZNJPC54LBN545BTYTWBH5OJT7DH35A, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545981719 (576h27m33.417064764s) [noise=true , address=145.79.227.52:4447 , piece_id=DCJSQWH5AUQQ466JPFVUZQJU2EISTTUB56XN3RY3QMJ3GJSYYT5Q, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.546368385 (576h27m33.417451421s) [noise=true , address=45.91.116.7:4011 , piece_id=SYKZRWQP7TXEXDXJVQAWZIEIX3B6AMT27XSHCFLEYIB2AA2T4TVQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.546711990 (576h27m33.417795035s) [noise=true , address=77.90.31.15:4697 , piece_id=7ZGRV7LGB6TR6UIJXDPCPBHXIGAA7RF3VWXQ5AOYRVUCHWNRDLGQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.546993867 (576h27m33.418076912s) [noise=true , address=45.91.116.5:4298 , piece_id=LMKURYTYLS7ZLFUDA7AY3BYPA234AEIHDMBVKRN5OPTD52CES6BQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.547300671 (576h27m33.418383706s) [noise=true , address=145.79.227.3:4150 , piece_id=457PQSGEFT6UKHV44ZFXEPRREYQOV6ILIWJWJCTQCBRXF7EIXXAA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.547598437 (576h27m33.418681472s) [noise=true , address=45.91.116.9:4261 , piece_id=RPNFXJQOUCXWVDOBLEOKYLEYV73QZYYYVTO7V5N3HPKIUQ2ZPN3Q, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.548170878 (576h27m33.419253913s) [noise=true , address=145.79.227.6:4259 , piece_id=BPYLWD3AMOC35GJPPC6E2VCKXIJX3OEEFJONLUYQRGFBES5NSNSA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.548496297 (576h27m33.419579332s) [noise=true , address=145.79.227.15:4873 , piece_id=E3DWN2MNIB4QQYUIFVWZLAHTI3OLQPKYRUREJOCUJLD555TL77NA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:35.244735136 (576h27m35.115818181s) [upload=505352]: Next returned nil segment
Dec 29 15:56:35.244750024 (576h27m35.115833079s) [upload=505352]: Waiting for error group managing segments...
Dec 29 15:56:35.528064171 (576h27m35.399147216s) [noise=true , address=145.79.227.5:4222 , piece_id=NOHTETEZOWG2FHCIUBH6T6FTCENFUICJZ7HKRSSP4SOKJDTXA5FQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.529417919 (576h27m35.400500964s) [noise=true , address=145.79.227.11:4730 , piece_id=OOZG5W4UTT4DFS2OKAFV2KD3KSV57GSSN5RPD6COULXEA7KIA45Q, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.540921563 (576h27m35.412004608s) [noise=true , address=145.79.227.6:4259 , piece_id=BPYLWD3AMOC35GJPPC6E2VCKXIJX3OEEFJONLUYQRGFBES5NSNSA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.546841225 (576h27m35.41792426s) [noise=true , address=145.79.227.15:4873 , piece_id=E3DWN2MNIB4QQYUIFVWZLAHTI3OLQPKYRUREJOCUJLD555TL77NA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.550607858 (576h27m35.421690903s) [noise=true , address=145.79.227.42:4121 , piece_id=SDH6PZRLR6HKXXQZ7QISEYZAZBPEP4HFBSAAZY23XVGLXJREJBMA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.562935632 (576h27m35.434018677s) [noise=true , address=145.79.227.52:4447 , piece_id=DCJSQWH5AUQQ466JPFVUZQJU2EISTTUB56XN3RY3QMJ3GJSYYT5Q, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.767765121 (576h27m35.638848166s) [noise=true , address=45.91.116.7:4011 , piece_id=SYKZRWQP7TXEXDXJVQAWZIEIX3B6AMT27XSHCFLEYIB2AA2T4TVQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.787755876 (576h27m35.658838911s) [noise=true , address=77.90.31.15:4697 , piece_id=7ZGRV7LGB6TR6UIJXDPCPBHXIGAA7RF3VWXQ5AOYRVUCHWNRDLGQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.807924060 (576h27m35.679007095s) [noise=true , address=77.90.31.18:4805 , piece_id=6HU7UEX2USEBWQ3RWPPKENRGL2TLC7BCT6UTXVRZEZQUTQBKHGLA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.829697982 (576h27m35.700781037s) [noise=true , address=77.90.31.14:4662 , piece_id=B67ICRUA7HPJFKS2DNDR4FQA2XHTH7CLZ7CPGNEAJULWW62W7JUA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.830640226 (576h27m35.701723271s) [noise=true , address=45.91.116.5:4298 , piece_id=LMKURYTYLS7ZLFUDA7AY3BYPA234AEIHDMBVKRN5OPTD52CES6BQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.910192497 (576h27m35.781275542s) [noise=true , address=45.91.116.12:4225 , piece_id=67TYC4MDAXRXFAJZC6TAJLZNJPC54LBN545BTYTWBH5OJT7DH35A, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:36.504292439 (576h27m36.375375484s) [noise=true , address=145.79.227.3:4150 , piece_id=457PQSGEFT6UKHV44ZFXEPRREYQOV6ILIWJWJCTQCBRXF7EIXXAA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:36.504319370 (576h27m36.375402416s) [seg_pos={0 0}, upload=505352]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:56:36.504544049 (576h27m36.375627094s) [noise=true , address=45.91.116.13:4156 , piece_id=WZGYZDPHV5EXGONPSF5FS57DOSO4CGTBB3GOLK5G3IN5NLRZ3NZQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 29 15:56:36.504552885 (576h27m36.37563592s) [noise=true , address=45.91.116.13:4156 , piece_id=WZGYZDPHV5EXGONPSF5FS57DOSO4CGTBB3GOLK5G3IN5NLRZ3NZQ, seg_pos={0 0}, upload=505352]: Long tail cancellation due to optimal threshold
Dec 29 15:56:36.504837718 (576h27m36.375920763s) [noise=true , address=45.91.116.8:4045 , piece_id=JCUDRBXV4SHGLFUMYCPAZXEN4C4T3HP6VXOMAO4Y3XJTLQX6IZVA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 29 15:56:36.504881811 (576h27m36.375964846s) [noise=true , address=45.91.116.8:4045 , piece_id=JCUDRBXV4SHGLFUMYCPAZXEN4C4T3HP6VXOMAO4Y3XJTLQX6IZVA, seg_pos={0 0}, upload=505352]: Long tail cancellation due to optimal threshold
Dec 29 15:56:36.504916918 (576h27m36.375999953s) [noise=true , address=45.91.116.9:4261 , piece_id=RPNFXJQOUCXWVDOBLEOKYLEYV73QZYYYVTO7V5N3HPKIUQ2ZPN3Q, seg_pos={0 0}, upload=505352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:47230->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:47230->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->45.91.116.9:4261: use of closed network connection
Dec 29 15:56:36.504927569 (576h27m36.376010614s) [noise=true , address=45.91.116.9:4261 , piece_id=RPNFXJQOUCXWVDOBLEOKYLEYV73QZYYYVTO7V5N3HPKIUQ2ZPN3Q, seg_pos={0 0}, upload=505352]: Long tail cancellation due to optimal threshold
Dec 29 15:56:36.505040294 (576h27m36.376123339s) [seg_pos={0 0}, upload=505352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:56:36.505185149 (576h27m36.376268184s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:56:36.505211309 (576h27m36.376294344s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:56:36.807637398 (576h27m36.678720473s) [upload=505352]: Done uploading segments...
Dec 29 15:56:36.807697783 (576h27m36.678780828s) [upload=505352]: Upload finished. err: <nil>
Duration: 5.760687784s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......X................X...................................] duration:
9.397464026s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 15:58:38 UTC)
orbiter warning: context deadline exceeded
Duration: 9.397614041s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [X..X.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XXXXXXXXXXXXXX.XXX] duration:
5.000836984s trace:
https://jaeger.dev.storj.io/trace/27d1960e82cb499a
FAIL (2025-12-29 15:58:26 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00086134s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [........X..........................XX.......................] duration:
30.386261437s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 15:58:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 30.3862779s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [X..X...XXX..XX.X..X..XX.XX...X.X..XX....XX.......X....X.X...] duration:
5.699599494s trace:
https://jaeger.dev.storj.io/trace/5a70c6bfa63ee1d5
FAIL (2025-12-29 15:58:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: LMKURYTYLS7ZLFUDA7AY3BYPA234AEIHDMBVKRN5OPTD52CES6BQ) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: 6HU7UEX2USEBWQ3RWPPKENRGL2TLC7BCT6UTXVRZEZQUTQBKHGLA) context deadline exceeded
======LOGS======
ec 29 15:56:33.292748031 (576h27m33.163831086s) [upload=505352]: Starting upload
Dec 29 15:56:33.292802995 (576h27m33.16388604s) [upload=505352]: Done starting upload
Dec 29 15:56:33.292870354 (576h27m33.163953399s) [upload=505352]: Uploading segments...
Dec 29 15:56:33.293052390 (576h27m33.164135425s) [upload=505352]: Got next segment. Inline: false
Dec 29 15:56:33.293081686 (576h27m33.164164721s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:56:33.293086184 (576h27m33.164169219s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:56:33.543074633 (576h27m33.414157678s) [seg_pos={0 0}, upload=505352]: Begin upload segment...
Dec 29 15:56:33.543129268 (576h27m33.414212303s) [seg_pos={0 0}, upload=505352]: Done begin upload segment.
Dec 29 15:56:33.543166018 (576h27m33.414249063s) [noise=true , address=145.79.227.5:4222 , piece_id=NOHTETEZOWG2FHCIUBH6T6FTCENFUICJZ7HKRSSP4SOKJDTXA5FQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.543223748 (576h27m33.414306793s) [noise=true , address=45.91.116.8:4045 , piece_id=JCUDRBXV4SHGLFUMYCPAZXEN4C4T3HP6VXOMAO4Y3XJTLQX6IZVA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.543852325 (576h27m33.41493537s) [noise=true , address=77.90.31.18:4805 , piece_id=6HU7UEX2USEBWQ3RWPPKENRGL2TLC7BCT6UTXVRZEZQUTQBKHGLA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.544288615 (576h27m33.41537166s) [noise=true , address=77.90.31.14:4662 , piece_id=B67ICRUA7HPJFKS2DNDR4FQA2XHTH7CLZ7CPGNEAJULWW62W7JUA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.544691412 (576h27m33.415774447s) [noise=true , address=145.79.227.42:4121 , piece_id=SDH6PZRLR6HKXXQZ7QISEYZAZBPEP4HFBSAAZY23XVGLXJREJBMA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545158792 (576h27m33.416241837s) [noise=true , address=45.91.116.13:4156 , piece_id=WZGYZDPHV5EXGONPSF5FS57DOSO4CGTBB3GOLK5G3IN5NLRZ3NZQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545530620 (576h27m33.416613675s) [noise=true , address=145.79.227.11:4730 , piece_id=OOZG5W4UTT4DFS2OKAFV2KD3KSV57GSSN5RPD6COULXEA7KIA45Q, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545966289 (576h27m33.417049334s) [noise=true , address=45.91.116.12:4225 , piece_id=67TYC4MDAXRXFAJZC6TAJLZNJPC54LBN545BTYTWBH5OJT7DH35A, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545981719 (576h27m33.417064764s) [noise=true , address=145.79.227.52:4447 , piece_id=DCJSQWH5AUQQ466JPFVUZQJU2EISTTUB56XN3RY3QMJ3GJSYYT5Q, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.546368385 (576h27m33.417451421s) [noise=true , address=45.91.116.7:4011 , piece_id=SYKZRWQP7TXEXDXJVQAWZIEIX3B6AMT27XSHCFLEYIB2AA2T4TVQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.546711990 (576h27m33.417795035s) [noise=true , address=77.90.31.15:4697 , piece_id=7ZGRV7LGB6TR6UIJXDPCPBHXIGAA7RF3VWXQ5AOYRVUCHWNRDLGQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.546993867 (576h27m33.418076912s) [noise=true , address=45.91.116.5:4298 , piece_id=LMKURYTYLS7ZLFUDA7AY3BYPA234AEIHDMBVKRN5OPTD52CES6BQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.547300671 (576h27m33.418383706s) [noise=true , address=145.79.227.3:4150 , piece_id=457PQSGEFT6UKHV44ZFXEPRREYQOV6ILIWJWJCTQCBRXF7EIXXAA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.547598437 (576h27m33.418681472s) [noise=true , address=45.91.116.9:4261 , piece_id=RPNFXJQOUCXWVDOBLEOKYLEYV73QZYYYVTO7V5N3HPKIUQ2ZPN3Q, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.548170878 (576h27m33.419253913s) [noise=true , address=145.79.227.6:4259 , piece_id=BPYLWD3AMOC35GJPPC6E2VCKXIJX3OEEFJONLUYQRGFBES5NSNSA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.548496297 (576h27m33.419579332s) [noise=true , address=145.79.227.15:4873 , piece_id=E3DWN2MNIB4QQYUIFVWZLAHTI3OLQPKYRUREJOCUJLD555TL77NA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:35.244735136 (576h27m35.115818181s) [upload=505352]: Next returned nil segment
Dec 29 15:56:35.244750024 (576h27m35.115833079s) [upload=505352]: Waiting for error group managing segments...
Dec 29 15:56:35.528064171 (576h27m35.399147216s) [noise=true , address=145.79.227.5:4222 , piece_id=NOHTETEZOWG2FHCIUBH6T6FTCENFUICJZ7HKRSSP4SOKJDTXA5FQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.529417919 (576h27m35.400500964s) [noise=true , address=145.79.227.11:4730 , piece_id=OOZG5W4UTT4DFS2OKAFV2KD3KSV57GSSN5RPD6COULXEA7KIA45Q, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.540921563 (576h27m35.412004608s) [noise=true , address=145.79.227.6:4259 , piece_id=BPYLWD3AMOC35GJPPC6E2VCKXIJX3OEEFJONLUYQRGFBES5NSNSA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.546841225 (576h27m35.41792426s) [noise=true , address=145.79.227.15:4873 , piece_id=E3DWN2MNIB4QQYUIFVWZLAHTI3OLQPKYRUREJOCUJLD555TL77NA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.550607858 (576h27m35.421690903s) [noise=true , address=145.79.227.42:4121 , piece_id=SDH6PZRLR6HKXXQZ7QISEYZAZBPEP4HFBSAAZY23XVGLXJREJBMA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.562935632 (576h27m35.434018677s) [noise=true , address=145.79.227.52:4447 , piece_id=DCJSQWH5AUQQ466JPFVUZQJU2EISTTUB56XN3RY3QMJ3GJSYYT5Q, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.767765121 (576h27m35.638848166s) [noise=true , address=45.91.116.7:4011 , piece_id=SYKZRWQP7TXEXDXJVQAWZIEIX3B6AMT27XSHCFLEYIB2AA2T4TVQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.787755876 (576h27m35.658838911s) [noise=true , address=77.90.31.15:4697 , piece_id=7ZGRV7LGB6TR6UIJXDPCPBHXIGAA7RF3VWXQ5AOYRVUCHWNRDLGQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.807924060 (576h27m35.679007095s) [noise=true , address=77.90.31.18:4805 , piece_id=6HU7UEX2USEBWQ3RWPPKENRGL2TLC7BCT6UTXVRZEZQUTQBKHGLA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.829697982 (576h27m35.700781037s) [noise=true , address=77.90.31.14:4662 , piece_id=B67ICRUA7HPJFKS2DNDR4FQA2XHTH7CLZ7CPGNEAJULWW62W7JUA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.830640226 (576h27m35.701723271s) [noise=true , address=45.91.116.5:4298 , piece_id=LMKURYTYLS7ZLFUDA7AY3BYPA234AEIHDMBVKRN5OPTD52CES6BQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.910192497 (576h27m35.781275542s) [noise=true , address=45.91.116.12:4225 , piece_id=67TYC4MDAXRXFAJZC6TAJLZNJPC54LBN545BTYTWBH5OJT7DH35A, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:36.504292439 (576h27m36.375375484s) [noise=true , address=145.79.227.3:4150 , piece_id=457PQSGEFT6UKHV44ZFXEPRREYQOV6ILIWJWJCTQCBRXF7EIXXAA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:36.504319370 (576h27m36.375402416s) [seg_pos={0 0}, upload=505352]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:56:36.504544049 (576h27m36.375627094s) [noise=true , address=45.91.116.13:4156 , piece_id=WZGYZDPHV5EXGONPSF5FS57DOSO4CGTBB3GOLK5G3IN5NLRZ3NZQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 29 15:56:36.504552885 (576h27m36.37563592s) [noise=true , address=45.91.116.13:4156 , piece_id=WZGYZDPHV5EXGONPSF5FS57DOSO4CGTBB3GOLK5G3IN5NLRZ3NZQ, seg_pos={0 0}, upload=505352]: Long tail cancellation due to optimal threshold
Dec 29 15:56:36.504837718 (576h27m36.375920763s) [noise=true , address=45.91.116.8:4045 , piece_id=JCUDRBXV4SHGLFUMYCPAZXEN4C4T3HP6VXOMAO4Y3XJTLQX6IZVA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 29 15:56:36.504881811 (576h27m36.375964846s) [noise=true , address=45.91.116.8:4045 , piece_id=JCUDRBXV4SHGLFUMYCPAZXEN4C4T3HP6VXOMAO4Y3XJTLQX6IZVA, seg_pos={0 0}, upload=505352]: Long tail cancellation due to optimal threshold
Dec 29 15:56:36.504916918 (576h27m36.375999953s) [noise=true , address=45.91.116.9:4261 , piece_id=RPNFXJQOUCXWVDOBLEOKYLEYV73QZYYYVTO7V5N3HPKIUQ2ZPN3Q, seg_pos={0 0}, upload=505352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:47230->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:47230->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->45.91.116.9:4261: use of closed network connection
Dec 29 15:56:36.504927569 (576h27m36.376010614s) [noise=true , address=45.91.116.9:4261 , piece_id=RPNFXJQOUCXWVDOBLEOKYLEYV73QZYYYVTO7V5N3HPKIUQ2ZPN3Q, seg_pos={0 0}, upload=505352]: Long tail cancellation due to optimal threshold
Dec 29 15:56:36.505040294 (576h27m36.376123339s) [seg_pos={0 0}, upload=505352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:56:36.505185149 (576h27m36.376268184s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:56:36.505211309 (576h27m36.376294344s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:56:36.807637398 (576h27m36.678720473s) [upload=505352]: Done uploading segments...
Dec 29 15:56:36.807697783 (576h27m36.678780828s) [upload=505352]: Upload finished. err: <nil>
Duration: 5.699625915s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........X................X..................................] duration:
15.186042012s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 15:58:13 UTC)
orbiter warning: context deadline exceeded
Duration: 15.186066099s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........X..........................XX......................] duration:
43.086338379s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 15:57:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 43.086354199s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [XX..X...XXX..XX.X..X..XX.XX...X.X..XX....XX.......X....X.X..] duration:
6.358874713s trace:
https://jaeger.dev.storj.io/trace/5a70c6bfa63ee1d5
FAIL (2025-12-29 15:57:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: LMKURYTYLS7ZLFUDA7AY3BYPA234AEIHDMBVKRN5OPTD52CES6BQ) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: 6HU7UEX2USEBWQ3RWPPKENRGL2TLC7BCT6UTXVRZEZQUTQBKHGLA) context deadline exceeded
======LOGS======
ec 29 15:56:33.292748031 (576h27m33.163831086s) [upload=505352]: Starting upload
Dec 29 15:56:33.292802995 (576h27m33.16388604s) [upload=505352]: Done starting upload
Dec 29 15:56:33.292870354 (576h27m33.163953399s) [upload=505352]: Uploading segments...
Dec 29 15:56:33.293052390 (576h27m33.164135425s) [upload=505352]: Got next segment. Inline: false
Dec 29 15:56:33.293081686 (576h27m33.164164721s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:56:33.293086184 (576h27m33.164169219s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:56:33.543074633 (576h27m33.414157678s) [seg_pos={0 0}, upload=505352]: Begin upload segment...
Dec 29 15:56:33.543129268 (576h27m33.414212303s) [seg_pos={0 0}, upload=505352]: Done begin upload segment.
Dec 29 15:56:33.543166018 (576h27m33.414249063s) [noise=true , address=145.79.227.5:4222 , piece_id=NOHTETEZOWG2FHCIUBH6T6FTCENFUICJZ7HKRSSP4SOKJDTXA5FQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.543223748 (576h27m33.414306793s) [noise=true , address=45.91.116.8:4045 , piece_id=JCUDRBXV4SHGLFUMYCPAZXEN4C4T3HP6VXOMAO4Y3XJTLQX6IZVA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.543852325 (576h27m33.41493537s) [noise=true , address=77.90.31.18:4805 , piece_id=6HU7UEX2USEBWQ3RWPPKENRGL2TLC7BCT6UTXVRZEZQUTQBKHGLA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.544288615 (576h27m33.41537166s) [noise=true , address=77.90.31.14:4662 , piece_id=B67ICRUA7HPJFKS2DNDR4FQA2XHTH7CLZ7CPGNEAJULWW62W7JUA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.544691412 (576h27m33.415774447s) [noise=true , address=145.79.227.42:4121 , piece_id=SDH6PZRLR6HKXXQZ7QISEYZAZBPEP4HFBSAAZY23XVGLXJREJBMA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545158792 (576h27m33.416241837s) [noise=true , address=45.91.116.13:4156 , piece_id=WZGYZDPHV5EXGONPSF5FS57DOSO4CGTBB3GOLK5G3IN5NLRZ3NZQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545530620 (576h27m33.416613675s) [noise=true , address=145.79.227.11:4730 , piece_id=OOZG5W4UTT4DFS2OKAFV2KD3KSV57GSSN5RPD6COULXEA7KIA45Q, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545966289 (576h27m33.417049334s) [noise=true , address=45.91.116.12:4225 , piece_id=67TYC4MDAXRXFAJZC6TAJLZNJPC54LBN545BTYTWBH5OJT7DH35A, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545981719 (576h27m33.417064764s) [noise=true , address=145.79.227.52:4447 , piece_id=DCJSQWH5AUQQ466JPFVUZQJU2EISTTUB56XN3RY3QMJ3GJSYYT5Q, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.546368385 (576h27m33.417451421s) [noise=true , address=45.91.116.7:4011 , piece_id=SYKZRWQP7TXEXDXJVQAWZIEIX3B6AMT27XSHCFLEYIB2AA2T4TVQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.546711990 (576h27m33.417795035s) [noise=true , address=77.90.31.15:4697 , piece_id=7ZGRV7LGB6TR6UIJXDPCPBHXIGAA7RF3VWXQ5AOYRVUCHWNRDLGQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.546993867 (576h27m33.418076912s) [noise=true , address=45.91.116.5:4298 , piece_id=LMKURYTYLS7ZLFUDA7AY3BYPA234AEIHDMBVKRN5OPTD52CES6BQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.547300671 (576h27m33.418383706s) [noise=true , address=145.79.227.3:4150 , piece_id=457PQSGEFT6UKHV44ZFXEPRREYQOV6ILIWJWJCTQCBRXF7EIXXAA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.547598437 (576h27m33.418681472s) [noise=true , address=45.91.116.9:4261 , piece_id=RPNFXJQOUCXWVDOBLEOKYLEYV73QZYYYVTO7V5N3HPKIUQ2ZPN3Q, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.548170878 (576h27m33.419253913s) [noise=true , address=145.79.227.6:4259 , piece_id=BPYLWD3AMOC35GJPPC6E2VCKXIJX3OEEFJONLUYQRGFBES5NSNSA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.548496297 (576h27m33.419579332s) [noise=true , address=145.79.227.15:4873 , piece_id=E3DWN2MNIB4QQYUIFVWZLAHTI3OLQPKYRUREJOCUJLD555TL77NA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:35.244735136 (576h27m35.115818181s) [upload=505352]: Next returned nil segment
Dec 29 15:56:35.244750024 (576h27m35.115833079s) [upload=505352]: Waiting for error group managing segments...
Dec 29 15:56:35.528064171 (576h27m35.399147216s) [noise=true , address=145.79.227.5:4222 , piece_id=NOHTETEZOWG2FHCIUBH6T6FTCENFUICJZ7HKRSSP4SOKJDTXA5FQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.529417919 (576h27m35.400500964s) [noise=true , address=145.79.227.11:4730 , piece_id=OOZG5W4UTT4DFS2OKAFV2KD3KSV57GSSN5RPD6COULXEA7KIA45Q, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.540921563 (576h27m35.412004608s) [noise=true , address=145.79.227.6:4259 , piece_id=BPYLWD3AMOC35GJPPC6E2VCKXIJX3OEEFJONLUYQRGFBES5NSNSA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.546841225 (576h27m35.41792426s) [noise=true , address=145.79.227.15:4873 , piece_id=E3DWN2MNIB4QQYUIFVWZLAHTI3OLQPKYRUREJOCUJLD555TL77NA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.550607858 (576h27m35.421690903s) [noise=true , address=145.79.227.42:4121 , piece_id=SDH6PZRLR6HKXXQZ7QISEYZAZBPEP4HFBSAAZY23XVGLXJREJBMA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.562935632 (576h27m35.434018677s) [noise=true , address=145.79.227.52:4447 , piece_id=DCJSQWH5AUQQ466JPFVUZQJU2EISTTUB56XN3RY3QMJ3GJSYYT5Q, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.767765121 (576h27m35.638848166s) [noise=true , address=45.91.116.7:4011 , piece_id=SYKZRWQP7TXEXDXJVQAWZIEIX3B6AMT27XSHCFLEYIB2AA2T4TVQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.787755876 (576h27m35.658838911s) [noise=true , address=77.90.31.15:4697 , piece_id=7ZGRV7LGB6TR6UIJXDPCPBHXIGAA7RF3VWXQ5AOYRVUCHWNRDLGQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.807924060 (576h27m35.679007095s) [noise=true , address=77.90.31.18:4805 , piece_id=6HU7UEX2USEBWQ3RWPPKENRGL2TLC7BCT6UTXVRZEZQUTQBKHGLA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.829697982 (576h27m35.700781037s) [noise=true , address=77.90.31.14:4662 , piece_id=B67ICRUA7HPJFKS2DNDR4FQA2XHTH7CLZ7CPGNEAJULWW62W7JUA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.830640226 (576h27m35.701723271s) [noise=true , address=45.91.116.5:4298 , piece_id=LMKURYTYLS7ZLFUDA7AY3BYPA234AEIHDMBVKRN5OPTD52CES6BQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.910192497 (576h27m35.781275542s) [noise=true , address=45.91.116.12:4225 , piece_id=67TYC4MDAXRXFAJZC6TAJLZNJPC54LBN545BTYTWBH5OJT7DH35A, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:36.504292439 (576h27m36.375375484s) [noise=true , address=145.79.227.3:4150 , piece_id=457PQSGEFT6UKHV44ZFXEPRREYQOV6ILIWJWJCTQCBRXF7EIXXAA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:36.504319370 (576h27m36.375402416s) [seg_pos={0 0}, upload=505352]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:56:36.504544049 (576h27m36.375627094s) [noise=true , address=45.91.116.13:4156 , piece_id=WZGYZDPHV5EXGONPSF5FS57DOSO4CGTBB3GOLK5G3IN5NLRZ3NZQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 29 15:56:36.504552885 (576h27m36.37563592s) [noise=true , address=45.91.116.13:4156 , piece_id=WZGYZDPHV5EXGONPSF5FS57DOSO4CGTBB3GOLK5G3IN5NLRZ3NZQ, seg_pos={0 0}, upload=505352]: Long tail cancellation due to optimal threshold
Dec 29 15:56:36.504837718 (576h27m36.375920763s) [noise=true , address=45.91.116.8:4045 , piece_id=JCUDRBXV4SHGLFUMYCPAZXEN4C4T3HP6VXOMAO4Y3XJTLQX6IZVA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 29 15:56:36.504881811 (576h27m36.375964846s) [noise=true , address=45.91.116.8:4045 , piece_id=JCUDRBXV4SHGLFUMYCPAZXEN4C4T3HP6VXOMAO4Y3XJTLQX6IZVA, seg_pos={0 0}, upload=505352]: Long tail cancellation due to optimal threshold
Dec 29 15:56:36.504916918 (576h27m36.375999953s) [noise=true , address=45.91.116.9:4261 , piece_id=RPNFXJQOUCXWVDOBLEOKYLEYV73QZYYYVTO7V5N3HPKIUQ2ZPN3Q, seg_pos={0 0}, upload=505352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:47230->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:47230->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->45.91.116.9:4261: use of closed network connection
Dec 29 15:56:36.504927569 (576h27m36.376010614s) [noise=true , address=45.91.116.9:4261 , piece_id=RPNFXJQOUCXWVDOBLEOKYLEYV73QZYYYVTO7V5N3HPKIUQ2ZPN3Q, seg_pos={0 0}, upload=505352]: Long tail cancellation due to optimal threshold
Dec 29 15:56:36.505040294 (576h27m36.376123339s) [seg_pos={0 0}, upload=505352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:56:36.505185149 (576h27m36.376268184s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:56:36.505211309 (576h27m36.376294344s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:56:36.807637398 (576h27m36.678720473s) [upload=505352]: Done uploading segments...
Dec 29 15:56:36.807697783 (576h27m36.678780828s) [upload=505352]: Upload finished. err: <nil>
Duration: 6.358903988s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [XX..X.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XXXXXXXXXXXXXX.XX] duration:
5.000984868s trace:
https://jaeger.dev.storj.io/trace/90d899abcf30539
FAIL (2025-12-29 15:57:24 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001009785s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [X.........X..........................XX.....................] duration:
44.658107048s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 15:57:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 44.658138968s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [XXX..X...XXX..XX.X..X..XX.XX...X.X..XX....XX.......X....X.X.] duration:
6.06725389s trace:
https://jaeger.dev.storj.io/trace/5a70c6bfa63ee1d5
FAIL (2025-12-29 15:57:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: LMKURYTYLS7ZLFUDA7AY3BYPA234AEIHDMBVKRN5OPTD52CES6BQ) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: 6HU7UEX2USEBWQ3RWPPKENRGL2TLC7BCT6UTXVRZEZQUTQBKHGLA) context deadline exceeded
======LOGS======
ec 29 15:56:33.292748031 (576h27m33.163831086s) [upload=505352]: Starting upload
Dec 29 15:56:33.292802995 (576h27m33.16388604s) [upload=505352]: Done starting upload
Dec 29 15:56:33.292870354 (576h27m33.163953399s) [upload=505352]: Uploading segments...
Dec 29 15:56:33.293052390 (576h27m33.164135425s) [upload=505352]: Got next segment. Inline: false
Dec 29 15:56:33.293081686 (576h27m33.164164721s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:56:33.293086184 (576h27m33.164169219s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:56:33.543074633 (576h27m33.414157678s) [seg_pos={0 0}, upload=505352]: Begin upload segment...
Dec 29 15:56:33.543129268 (576h27m33.414212303s) [seg_pos={0 0}, upload=505352]: Done begin upload segment.
Dec 29 15:56:33.543166018 (576h27m33.414249063s) [noise=true , address=145.79.227.5:4222 , piece_id=NOHTETEZOWG2FHCIUBH6T6FTCENFUICJZ7HKRSSP4SOKJDTXA5FQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.543223748 (576h27m33.414306793s) [noise=true , address=45.91.116.8:4045 , piece_id=JCUDRBXV4SHGLFUMYCPAZXEN4C4T3HP6VXOMAO4Y3XJTLQX6IZVA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.543852325 (576h27m33.41493537s) [noise=true , address=77.90.31.18:4805 , piece_id=6HU7UEX2USEBWQ3RWPPKENRGL2TLC7BCT6UTXVRZEZQUTQBKHGLA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.544288615 (576h27m33.41537166s) [noise=true , address=77.90.31.14:4662 , piece_id=B67ICRUA7HPJFKS2DNDR4FQA2XHTH7CLZ7CPGNEAJULWW62W7JUA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.544691412 (576h27m33.415774447s) [noise=true , address=145.79.227.42:4121 , piece_id=SDH6PZRLR6HKXXQZ7QISEYZAZBPEP4HFBSAAZY23XVGLXJREJBMA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545158792 (576h27m33.416241837s) [noise=true , address=45.91.116.13:4156 , piece_id=WZGYZDPHV5EXGONPSF5FS57DOSO4CGTBB3GOLK5G3IN5NLRZ3NZQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545530620 (576h27m33.416613675s) [noise=true , address=145.79.227.11:4730 , piece_id=OOZG5W4UTT4DFS2OKAFV2KD3KSV57GSSN5RPD6COULXEA7KIA45Q, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545966289 (576h27m33.417049334s) [noise=true , address=45.91.116.12:4225 , piece_id=67TYC4MDAXRXFAJZC6TAJLZNJPC54LBN545BTYTWBH5OJT7DH35A, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545981719 (576h27m33.417064764s) [noise=true , address=145.79.227.52:4447 , piece_id=DCJSQWH5AUQQ466JPFVUZQJU2EISTTUB56XN3RY3QMJ3GJSYYT5Q, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.546368385 (576h27m33.417451421s) [noise=true , address=45.91.116.7:4011 , piece_id=SYKZRWQP7TXEXDXJVQAWZIEIX3B6AMT27XSHCFLEYIB2AA2T4TVQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.546711990 (576h27m33.417795035s) [noise=true , address=77.90.31.15:4697 , piece_id=7ZGRV7LGB6TR6UIJXDPCPBHXIGAA7RF3VWXQ5AOYRVUCHWNRDLGQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.546993867 (576h27m33.418076912s) [noise=true , address=45.91.116.5:4298 , piece_id=LMKURYTYLS7ZLFUDA7AY3BYPA234AEIHDMBVKRN5OPTD52CES6BQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.547300671 (576h27m33.418383706s) [noise=true , address=145.79.227.3:4150 , piece_id=457PQSGEFT6UKHV44ZFXEPRREYQOV6ILIWJWJCTQCBRXF7EIXXAA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.547598437 (576h27m33.418681472s) [noise=true , address=45.91.116.9:4261 , piece_id=RPNFXJQOUCXWVDOBLEOKYLEYV73QZYYYVTO7V5N3HPKIUQ2ZPN3Q, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.548170878 (576h27m33.419253913s) [noise=true , address=145.79.227.6:4259 , piece_id=BPYLWD3AMOC35GJPPC6E2VCKXIJX3OEEFJONLUYQRGFBES5NSNSA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.548496297 (576h27m33.419579332s) [noise=true , address=145.79.227.15:4873 , piece_id=E3DWN2MNIB4QQYUIFVWZLAHTI3OLQPKYRUREJOCUJLD555TL77NA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:35.244735136 (576h27m35.115818181s) [upload=505352]: Next returned nil segment
Dec 29 15:56:35.244750024 (576h27m35.115833079s) [upload=505352]: Waiting for error group managing segments...
Dec 29 15:56:35.528064171 (576h27m35.399147216s) [noise=true , address=145.79.227.5:4222 , piece_id=NOHTETEZOWG2FHCIUBH6T6FTCENFUICJZ7HKRSSP4SOKJDTXA5FQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.529417919 (576h27m35.400500964s) [noise=true , address=145.79.227.11:4730 , piece_id=OOZG5W4UTT4DFS2OKAFV2KD3KSV57GSSN5RPD6COULXEA7KIA45Q, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.540921563 (576h27m35.412004608s) [noise=true , address=145.79.227.6:4259 , piece_id=BPYLWD3AMOC35GJPPC6E2VCKXIJX3OEEFJONLUYQRGFBES5NSNSA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.546841225 (576h27m35.41792426s) [noise=true , address=145.79.227.15:4873 , piece_id=E3DWN2MNIB4QQYUIFVWZLAHTI3OLQPKYRUREJOCUJLD555TL77NA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.550607858 (576h27m35.421690903s) [noise=true , address=145.79.227.42:4121 , piece_id=SDH6PZRLR6HKXXQZ7QISEYZAZBPEP4HFBSAAZY23XVGLXJREJBMA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.562935632 (576h27m35.434018677s) [noise=true , address=145.79.227.52:4447 , piece_id=DCJSQWH5AUQQ466JPFVUZQJU2EISTTUB56XN3RY3QMJ3GJSYYT5Q, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.767765121 (576h27m35.638848166s) [noise=true , address=45.91.116.7:4011 , piece_id=SYKZRWQP7TXEXDXJVQAWZIEIX3B6AMT27XSHCFLEYIB2AA2T4TVQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.787755876 (576h27m35.658838911s) [noise=true , address=77.90.31.15:4697 , piece_id=7ZGRV7LGB6TR6UIJXDPCPBHXIGAA7RF3VWXQ5AOYRVUCHWNRDLGQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.807924060 (576h27m35.679007095s) [noise=true , address=77.90.31.18:4805 , piece_id=6HU7UEX2USEBWQ3RWPPKENRGL2TLC7BCT6UTXVRZEZQUTQBKHGLA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.829697982 (576h27m35.700781037s) [noise=true , address=77.90.31.14:4662 , piece_id=B67ICRUA7HPJFKS2DNDR4FQA2XHTH7CLZ7CPGNEAJULWW62W7JUA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.830640226 (576h27m35.701723271s) [noise=true , address=45.91.116.5:4298 , piece_id=LMKURYTYLS7ZLFUDA7AY3BYPA234AEIHDMBVKRN5OPTD52CES6BQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.910192497 (576h27m35.781275542s) [noise=true , address=45.91.116.12:4225 , piece_id=67TYC4MDAXRXFAJZC6TAJLZNJPC54LBN545BTYTWBH5OJT7DH35A, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:36.504292439 (576h27m36.375375484s) [noise=true , address=145.79.227.3:4150 , piece_id=457PQSGEFT6UKHV44ZFXEPRREYQOV6ILIWJWJCTQCBRXF7EIXXAA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:36.504319370 (576h27m36.375402416s) [seg_pos={0 0}, upload=505352]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:56:36.504544049 (576h27m36.375627094s) [noise=true , address=45.91.116.13:4156 , piece_id=WZGYZDPHV5EXGONPSF5FS57DOSO4CGTBB3GOLK5G3IN5NLRZ3NZQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 29 15:56:36.504552885 (576h27m36.37563592s) [noise=true , address=45.91.116.13:4156 , piece_id=WZGYZDPHV5EXGONPSF5FS57DOSO4CGTBB3GOLK5G3IN5NLRZ3NZQ, seg_pos={0 0}, upload=505352]: Long tail cancellation due to optimal threshold
Dec 29 15:56:36.504837718 (576h27m36.375920763s) [noise=true , address=45.91.116.8:4045 , piece_id=JCUDRBXV4SHGLFUMYCPAZXEN4C4T3HP6VXOMAO4Y3XJTLQX6IZVA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 29 15:56:36.504881811 (576h27m36.375964846s) [noise=true , address=45.91.116.8:4045 , piece_id=JCUDRBXV4SHGLFUMYCPAZXEN4C4T3HP6VXOMAO4Y3XJTLQX6IZVA, seg_pos={0 0}, upload=505352]: Long tail cancellation due to optimal threshold
Dec 29 15:56:36.504916918 (576h27m36.375999953s) [noise=true , address=45.91.116.9:4261 , piece_id=RPNFXJQOUCXWVDOBLEOKYLEYV73QZYYYVTO7V5N3HPKIUQ2ZPN3Q, seg_pos={0 0}, upload=505352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:47230->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:47230->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->45.91.116.9:4261: use of closed network connection
Dec 29 15:56:36.504927569 (576h27m36.376010614s) [noise=true , address=45.91.116.9:4261 , piece_id=RPNFXJQOUCXWVDOBLEOKYLEYV73QZYYYVTO7V5N3HPKIUQ2ZPN3Q, seg_pos={0 0}, upload=505352]: Long tail cancellation due to optimal threshold
Dec 29 15:56:36.505040294 (576h27m36.376123339s) [seg_pos={0 0}, upload=505352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:56:36.505185149 (576h27m36.376268184s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:56:36.505211309 (576h27m36.376294344s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:56:36.807637398 (576h27m36.678720473s) [upload=505352]: Done uploading segments...
Dec 29 15:56:36.807697783 (576h27m36.678780828s) [upload=505352]: Upload finished. err: <nil>
Duration: 6.06727463s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........X................X.................................] duration:
13.855333692s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 15:57:08 UTC)
orbiter warning: context deadline exceeded
Duration: 13.855352319s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [.XXX..X...XXX..XX.X..X..XX.XX...X.X..XX....XX.......X....X.X] duration:
30.010752252s trace:
https://jaeger.dev.storj.io/trace/5a70c6bfa63ee1d5
FAIL (2025-12-29 15:57:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: LMKURYTYLS7ZLFUDA7AY3BYPA234AEIHDMBVKRN5OPTD52CES6BQ) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: 6HU7UEX2USEBWQ3RWPPKENRGL2TLC7BCT6UTXVRZEZQUTQBKHGLA) context deadline exceeded
======LOGS======
ec 29 15:56:33.292748031 (576h27m33.163831086s) [upload=505352]: Starting upload
Dec 29 15:56:33.292802995 (576h27m33.16388604s) [upload=505352]: Done starting upload
Dec 29 15:56:33.292870354 (576h27m33.163953399s) [upload=505352]: Uploading segments...
Dec 29 15:56:33.293052390 (576h27m33.164135425s) [upload=505352]: Got next segment. Inline: false
Dec 29 15:56:33.293081686 (576h27m33.164164721s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:56:33.293086184 (576h27m33.164169219s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:56:33.543074633 (576h27m33.414157678s) [seg_pos={0 0}, upload=505352]: Begin upload segment...
Dec 29 15:56:33.543129268 (576h27m33.414212303s) [seg_pos={0 0}, upload=505352]: Done begin upload segment.
Dec 29 15:56:33.543166018 (576h27m33.414249063s) [noise=true , address=145.79.227.5:4222 , piece_id=NOHTETEZOWG2FHCIUBH6T6FTCENFUICJZ7HKRSSP4SOKJDTXA5FQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.543223748 (576h27m33.414306793s) [noise=true , address=45.91.116.8:4045 , piece_id=JCUDRBXV4SHGLFUMYCPAZXEN4C4T3HP6VXOMAO4Y3XJTLQX6IZVA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.543852325 (576h27m33.41493537s) [noise=true , address=77.90.31.18:4805 , piece_id=6HU7UEX2USEBWQ3RWPPKENRGL2TLC7BCT6UTXVRZEZQUTQBKHGLA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.544288615 (576h27m33.41537166s) [noise=true , address=77.90.31.14:4662 , piece_id=B67ICRUA7HPJFKS2DNDR4FQA2XHTH7CLZ7CPGNEAJULWW62W7JUA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.544691412 (576h27m33.415774447s) [noise=true , address=145.79.227.42:4121 , piece_id=SDH6PZRLR6HKXXQZ7QISEYZAZBPEP4HFBSAAZY23XVGLXJREJBMA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545158792 (576h27m33.416241837s) [noise=true , address=45.91.116.13:4156 , piece_id=WZGYZDPHV5EXGONPSF5FS57DOSO4CGTBB3GOLK5G3IN5NLRZ3NZQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545530620 (576h27m33.416613675s) [noise=true , address=145.79.227.11:4730 , piece_id=OOZG5W4UTT4DFS2OKAFV2KD3KSV57GSSN5RPD6COULXEA7KIA45Q, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545966289 (576h27m33.417049334s) [noise=true , address=45.91.116.12:4225 , piece_id=67TYC4MDAXRXFAJZC6TAJLZNJPC54LBN545BTYTWBH5OJT7DH35A, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.545981719 (576h27m33.417064764s) [noise=true , address=145.79.227.52:4447 , piece_id=DCJSQWH5AUQQ466JPFVUZQJU2EISTTUB56XN3RY3QMJ3GJSYYT5Q, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.546368385 (576h27m33.417451421s) [noise=true , address=45.91.116.7:4011 , piece_id=SYKZRWQP7TXEXDXJVQAWZIEIX3B6AMT27XSHCFLEYIB2AA2T4TVQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.546711990 (576h27m33.417795035s) [noise=true , address=77.90.31.15:4697 , piece_id=7ZGRV7LGB6TR6UIJXDPCPBHXIGAA7RF3VWXQ5AOYRVUCHWNRDLGQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.546993867 (576h27m33.418076912s) [noise=true , address=45.91.116.5:4298 , piece_id=LMKURYTYLS7ZLFUDA7AY3BYPA234AEIHDMBVKRN5OPTD52CES6BQ, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.547300671 (576h27m33.418383706s) [noise=true , address=145.79.227.3:4150 , piece_id=457PQSGEFT6UKHV44ZFXEPRREYQOV6ILIWJWJCTQCBRXF7EIXXAA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.547598437 (576h27m33.418681472s) [noise=true , address=45.91.116.9:4261 , piece_id=RPNFXJQOUCXWVDOBLEOKYLEYV73QZYYYVTO7V5N3HPKIUQ2ZPN3Q, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.548170878 (576h27m33.419253913s) [noise=true , address=145.79.227.6:4259 , piece_id=BPYLWD3AMOC35GJPPC6E2VCKXIJX3OEEFJONLUYQRGFBES5NSNSA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:33.548496297 (576h27m33.419579332s) [noise=true , address=145.79.227.15:4873 , piece_id=E3DWN2MNIB4QQYUIFVWZLAHTI3OLQPKYRUREJOCUJLD555TL77NA, seg_pos={0 0}, upload=505352]: Uploading piece...
Dec 29 15:56:35.244735136 (576h27m35.115818181s) [upload=505352]: Next returned nil segment
Dec 29 15:56:35.244750024 (576h27m35.115833079s) [upload=505352]: Waiting for error group managing segments...
Dec 29 15:56:35.528064171 (576h27m35.399147216s) [noise=true , address=145.79.227.5:4222 , piece_id=NOHTETEZOWG2FHCIUBH6T6FTCENFUICJZ7HKRSSP4SOKJDTXA5FQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.529417919 (576h27m35.400500964s) [noise=true , address=145.79.227.11:4730 , piece_id=OOZG5W4UTT4DFS2OKAFV2KD3KSV57GSSN5RPD6COULXEA7KIA45Q, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.540921563 (576h27m35.412004608s) [noise=true , address=145.79.227.6:4259 , piece_id=BPYLWD3AMOC35GJPPC6E2VCKXIJX3OEEFJONLUYQRGFBES5NSNSA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.546841225 (576h27m35.41792426s) [noise=true , address=145.79.227.15:4873 , piece_id=E3DWN2MNIB4QQYUIFVWZLAHTI3OLQPKYRUREJOCUJLD555TL77NA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.550607858 (576h27m35.421690903s) [noise=true , address=145.79.227.42:4121 , piece_id=SDH6PZRLR6HKXXQZ7QISEYZAZBPEP4HFBSAAZY23XVGLXJREJBMA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.562935632 (576h27m35.434018677s) [noise=true , address=145.79.227.52:4447 , piece_id=DCJSQWH5AUQQ466JPFVUZQJU2EISTTUB56XN3RY3QMJ3GJSYYT5Q, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.767765121 (576h27m35.638848166s) [noise=true , address=45.91.116.7:4011 , piece_id=SYKZRWQP7TXEXDXJVQAWZIEIX3B6AMT27XSHCFLEYIB2AA2T4TVQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.787755876 (576h27m35.658838911s) [noise=true , address=77.90.31.15:4697 , piece_id=7ZGRV7LGB6TR6UIJXDPCPBHXIGAA7RF3VWXQ5AOYRVUCHWNRDLGQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.807924060 (576h27m35.679007095s) [noise=true , address=77.90.31.18:4805 , piece_id=6HU7UEX2USEBWQ3RWPPKENRGL2TLC7BCT6UTXVRZEZQUTQBKHGLA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.829697982 (576h27m35.700781037s) [noise=true , address=77.90.31.14:4662 , piece_id=B67ICRUA7HPJFKS2DNDR4FQA2XHTH7CLZ7CPGNEAJULWW62W7JUA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.830640226 (576h27m35.701723271s) [noise=true , address=45.91.116.5:4298 , piece_id=LMKURYTYLS7ZLFUDA7AY3BYPA234AEIHDMBVKRN5OPTD52CES6BQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:35.910192497 (576h27m35.781275542s) [noise=true , address=45.91.116.12:4225 , piece_id=67TYC4MDAXRXFAJZC6TAJLZNJPC54LBN545BTYTWBH5OJT7DH35A, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:36.504292439 (576h27m36.375375484s) [noise=true , address=145.79.227.3:4150 , piece_id=457PQSGEFT6UKHV44ZFXEPRREYQOV6ILIWJWJCTQCBRXF7EIXXAA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: <nil>
Dec 29 15:56:36.504319370 (576h27m36.375402416s) [seg_pos={0 0}, upload=505352]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:56:36.504544049 (576h27m36.375627094s) [noise=true , address=45.91.116.13:4156 , piece_id=WZGYZDPHV5EXGONPSF5FS57DOSO4CGTBB3GOLK5G3IN5NLRZ3NZQ, seg_pos={0 0}, upload=505352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 29 15:56:36.504552885 (576h27m36.37563592s) [noise=true , address=45.91.116.13:4156 , piece_id=WZGYZDPHV5EXGONPSF5FS57DOSO4CGTBB3GOLK5G3IN5NLRZ3NZQ, seg_pos={0 0}, upload=505352]: Long tail cancellation due to optimal threshold
Dec 29 15:56:36.504837718 (576h27m36.375920763s) [noise=true , address=45.91.116.8:4045 , piece_id=JCUDRBXV4SHGLFUMYCPAZXEN4C4T3HP6VXOMAO4Y3XJTLQX6IZVA, seg_pos={0 0}, upload=505352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 29 15:56:36.504881811 (576h27m36.375964846s) [noise=true , address=45.91.116.8:4045 , piece_id=JCUDRBXV4SHGLFUMYCPAZXEN4C4T3HP6VXOMAO4Y3XJTLQX6IZVA, seg_pos={0 0}, upload=505352]: Long tail cancellation due to optimal threshold
Dec 29 15:56:36.504916918 (576h27m36.375999953s) [noise=true , address=45.91.116.9:4261 , piece_id=RPNFXJQOUCXWVDOBLEOKYLEYV73QZYYYVTO7V5N3HPKIUQ2ZPN3Q, seg_pos={0 0}, upload=505352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:47230->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:47230->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->45.91.116.9:4261: use of closed network connection
Dec 29 15:56:36.504927569 (576h27m36.376010614s) [noise=true , address=45.91.116.9:4261 , piece_id=RPNFXJQOUCXWVDOBLEOKYLEYV73QZYYYVTO7V5N3HPKIUQ2ZPN3Q, seg_pos={0 0}, upload=505352]: Long tail cancellation due to optimal threshold
Dec 29 15:56:36.505040294 (576h27m36.376123339s) [seg_pos={0 0}, upload=505352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:56:36.505185149 (576h27m36.376268184s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:56:36.505211309 (576h27m36.376294344s) [upload=505352]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:56:36.807637398 (576h27m36.678720473s) [upload=505352]: Done uploading segments...
Dec 29 15:56:36.807697783 (576h27m36.678780828s) [upload=505352]: Upload finished. err: <nil>
Duration: 30.010775517s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [XXX..X.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XXXXXXXXXXXXXX.X] duration:
5.001420967s trace:
https://jaeger.dev.storj.io/trace/31e59e628039baae
FAIL (2025-12-29 15:56:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001446636s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X.........X..........................XX....................] duration:
44.338401869s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 15:56:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 44.33842825s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [.XXX..X.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XXXXXXXXXXXXXX.] duration:
4.840335747s trace:
https://jaeger.dev.storj.io/trace/24c765efc620c96f
FAIL (2025-12-29 15:56:12 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.840364911s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........X................X................................] duration:
16.16265855s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 15:56:10 UTC)
orbiter warning: context deadline exceeded
Duration: 16.162684238s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [X.XXX..X...XXX..XX.X..X..XX.XX...X.X..XX....XX.......X....X.] duration:
6.017930919s trace:
https://jaeger.dev.storj.io/trace/75bbd55714594abb
FAIL (2025-12-29 15:56:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: S7CYPEJHQFCO2LZZS6ZIA6CH32WMJ7AMVJLREBKZO6IMRN2CVCDQ) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: FEIKCTMJJCRG22CS76TOQPX3FMHJ4OLPTS3MYAFJSGSY54S46XNA) context deadline exceeded
======LOGS======
Dec 29 15:55:29.640180914 (576h26m29.511263949s) [upload=505337]: Starting upload
Dec 29 15:55:29.640266207 (576h26m29.511349252s) [upload=505337]: Done starting upload
Dec 29 15:55:29.640406164 (576h26m29.511489209s) [upload=505337]: Uploading segments...
Dec 29 15:55:29.640519690 (576h26m29.511602735s) [upload=505337]: Got next segment. Inline: false
Dec 29 15:55:29.640571137 (576h26m29.511654172s) [upload=505337]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:55:29.640578492 (576h26m29.511661527s) [upload=505337]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:55:29.889648322 (576h26m29.760731367s) [seg_pos={0 0}, upload=505337]: Begin upload segment...
Dec 29 15:55:29.889701363 (576h26m29.760784408s) [seg_pos={0 0}, upload=505337]: Done begin upload segment.
Dec 29 15:55:29.889731862 (576h26m29.760814907s) [noise=true , address=145.79.227.45:4229 , piece_id=MIPS27E3X5NLANE74YFEZO3WKZUZJVHOTSEPGG4A5I6UFBXH5SAA, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.889829578 (576h26m29.760912613s) [noise=true , address=45.91.116.5:4298 , piece_id=BZPEIKV3K7LQ2KZ7MZZHCMSLEQM3UQFBB4OH2WXL6J3AF7LA3CKA, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.890278001 (576h26m29.761361046s) [noise=true , address=145.79.227.13:4804 , piece_id=PJWSRBMF3E5LKULZPROZ4EUJETAXHJBU2YTR2SDUM75GGJWEDQTQ, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.890606848 (576h26m29.761689883s) [noise=true , address=45.91.116.4:4939 , piece_id=CIEFNVNHT7LGC5SAF2Z5EPTXWZFROXWD3GAMPNGRS3AHWA3IUTQQ, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.890882402 (576h26m29.761965437s) [noise=true , address=145.79.227.5:4221 , piece_id=I45HGGEEZXUXAXLXAPWD6LQBW2P2X3JHYIEDUXHUPXVWNOW7BQUQ, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.891140964 (576h26m29.762223999s) [noise=true , address=45.91.116.6:4975 , piece_id=HSBQ4SUC5FHXUGITJYFLWO7HBZRYMXQNPQUZQH2I5GS5TUBFDVNA, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.891501762 (576h26m29.762584797s) [noise=true , address=145.79.227.12:4765 , piece_id=X3KOZ2UNK5Y2WLUXHEP5W6RBYXHHIISETW5H7RA25APPA75J377A, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.891829666 (576h26m29.762912701s) [noise=true , address=145.79.227.19:4078 , piece_id=4UNDCN5P72AOBJ5XFROMZDAM2GQMLK7GX5LQQAQ42L2B2TRE36GQ, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.892207926 (576h26m29.763290961s) [noise=true , address=145.79.227.47:4265 , piece_id=XV2PUO5DYJZ5IM6NMYRZF7RKZTEEIPKUMMP3PRQ4JFGFQMCYTM3A, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.894028833 (576h26m29.765111878s) [noise=true , address=145.79.227.39:4013 , piece_id=2ODPDKVUOIYOTUUVGM7SRXBFKF6CYC6GOAI24U7E5CLH35ILQRGQ, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.895792131 (576h26m29.766875176s) [noise=true , address=145.79.227.18:4006 , piece_id=HWDTN7APQXFULHEQJ6M3FWIX6KEZYGJVKGCRDI6CYMJ6F4MZF3QQ, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.897572311 (576h26m29.768655346s) [noise=true , address=45.91.116.14:4192 , piece_id=S7CYPEJHQFCO2LZZS6ZIA6CH32WMJ7AMVJLREBKZO6IMRN2CVCDQ, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.898843241 (576h26m29.769926287s) [noise=true , address=145.79.227.49:4337 , piece_id=CJO6AX7KMJJBAGIO5NJ63Z4LUJYH4DMNWMFCLTFAFUMLQDJHC5YA, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.900004663 (576h26m29.771087698s) [noise=true , address=45.91.116.11:4119 , piece_id=ZDXKSDGVQPOZX5SS7ANND6XAQJXZFF7UBCWFTUPGCHNSFHIEW4KA, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.901217452 (576h26m29.772300487s) [noise=true , address=145.79.227.6:4257 , piece_id=4WHKFOSFQSBKMDDCOWU45KJKKDDPYFBFVWWBNJASD44CLGAAXOYQ, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.902264686 (576h26m29.773347721s) [noise=true , address=77.90.31.14:4661 , piece_id=FEIKCTMJJCRG22CS76TOQPX3FMHJ4OLPTS3MYAFJSGSY54S46XNA, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:31.592207896 (576h26m31.463290941s) [upload=505337]: Next returned nil segment
Dec 29 15:55:31.592224318 (576h26m31.463307353s) [upload=505337]: Waiting for error group managing segments...
Dec 29 15:55:31.873758185 (576h26m31.74484122s) [noise=true , address=145.79.227.45:4229 , piece_id=MIPS27E3X5NLANE74YFEZO3WKZUZJVHOTSEPGG4A5I6UFBXH5SAA, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:31.875984494 (576h26m31.747067529s) [noise=true , address=145.79.227.5:4221 , piece_id=I45HGGEEZXUXAXLXAPWD6LQBW2P2X3JHYIEDUXHUPXVWNOW7BQUQ, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:31.876707992 (576h26m31.747791037s) [noise=true , address=145.79.227.6:4257 , piece_id=4WHKFOSFQSBKMDDCOWU45KJKKDDPYFBFVWWBNJASD44CLGAAXOYQ, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:31.889903798 (576h26m31.760986833s) [noise=true , address=145.79.227.18:4006 , piece_id=HWDTN7APQXFULHEQJ6M3FWIX6KEZYGJVKGCRDI6CYMJ6F4MZF3QQ, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:31.890572651 (576h26m31.761655686s) [noise=true , address=145.79.227.19:4078 , piece_id=4UNDCN5P72AOBJ5XFROMZDAM2GQMLK7GX5LQQAQ42L2B2TRE36GQ, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:31.895589305 (576h26m31.76667235s) [noise=true , address=145.79.227.13:4804 , piece_id=PJWSRBMF3E5LKULZPROZ4EUJETAXHJBU2YTR2SDUM75GGJWEDQTQ, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:31.900771702 (576h26m31.771854747s) [noise=true , address=145.79.227.12:4765 , piece_id=X3KOZ2UNK5Y2WLUXHEP5W6RBYXHHIISETW5H7RA25APPA75J377A, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:31.903510758 (576h26m31.774593813s) [noise=true , address=145.79.227.47:4265 , piece_id=XV2PUO5DYJZ5IM6NMYRZF7RKZTEEIPKUMMP3PRQ4JFGFQMCYTM3A, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:31.909782751 (576h26m31.780865796s) [noise=true , address=145.79.227.39:4013 , piece_id=2ODPDKVUOIYOTUUVGM7SRXBFKF6CYC6GOAI24U7E5CLH35ILQRGQ, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:32.167331647 (576h26m32.038414692s) [noise=true , address=45.91.116.5:4298 , piece_id=BZPEIKV3K7LQ2KZ7MZZHCMSLEQM3UQFBB4OH2WXL6J3AF7LA3CKA, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:32.618463797 (576h26m32.489546842s) [noise=true , address=45.91.116.11:4119 , piece_id=ZDXKSDGVQPOZX5SS7ANND6XAQJXZFF7UBCWFTUPGCHNSFHIEW4KA, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:32.854132278 (576h26m32.725215323s) [noise=true , address=77.90.31.14:4661 , piece_id=FEIKCTMJJCRG22CS76TOQPX3FMHJ4OLPTS3MYAFJSGSY54S46XNA, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:32.856202470 (576h26m32.727285505s) [noise=true , address=45.91.116.14:4192 , piece_id=S7CYPEJHQFCO2LZZS6ZIA6CH32WMJ7AMVJLREBKZO6IMRN2CVCDQ, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:32.856220114 (576h26m32.727303149s) [seg_pos={0 0}, upload=505337]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:55:32.856560972 (576h26m32.727644017s) [noise=true , address=45.91.116.6:4975 , piece_id=HSBQ4SUC5FHXUGITJYFLWO7HBZRYMXQNPQUZQH2I5GS5TUBFDVNA, seg_pos={0 0}, upload=505337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 29 15:55:32.856591049 (576h26m32.727674094s) [noise=true , address=45.91.116.6:4975 , piece_id=HSBQ4SUC5FHXUGITJYFLWO7HBZRYMXQNPQUZQH2I5GS5TUBFDVNA, seg_pos={0 0}, upload=505337]: Long tail cancellation due to optimal threshold
Dec 29 15:55:32.856630064 (576h26m32.727713109s) [noise=true , address=45.91.116.4:4939 , piece_id=CIEFNVNHT7LGC5SAF2Z5EPTXWZFROXWD3GAMPNGRS3AHWA3IUTQQ, seg_pos={0 0}, upload=505337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39800->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 29 15:55:32.856639723 (576h26m32.727722768s) [noise=true , address=45.91.116.4:4939 , piece_id=CIEFNVNHT7LGC5SAF2Z5EPTXWZFROXWD3GAMPNGRS3AHWA3IUTQQ, seg_pos={0 0}, upload=505337]: Long tail cancellation due to optimal threshold
Dec 29 15:55:32.856756956 (576h26m32.727839991s) [noise=true , address=145.79.227.49:4337 , piece_id=CJO6AX7KMJJBAGIO5NJ63Z4LUJYH4DMNWMFCLTFAFUMLQDJHC5YA, seg_pos={0 0}, upload=505337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF
Dec 29 15:55:32.856764610 (576h26m32.727847645s) [noise=true , address=145.79.227.49:4337 , piece_id=CJO6AX7KMJJBAGIO5NJ63Z4LUJYH4DMNWMFCLTFAFUMLQDJHC5YA, seg_pos={0 0}, upload=505337]: Long tail cancellation due to optimal threshold
Dec 29 15:55:32.856784769 (576h26m32.727867804s) [seg_pos={0 0}, upload=505337]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:55:32.856894067 (576h26m32.727977112s) [upload=505337]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:55:32.856911990 (576h26m32.727995035s) [upload=505337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:55:33.153684143 (576h26m33.024767178s) [upload=505337]: Done uploading segments...
Dec 29 15:55:33.153705995 (576h26m33.02478903s) [upload=505337]: Upload finished. err: <nil>
Duration: 6.017959514s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [XX.XXX..X...XXX..XX.X..X..XX.XX...X.X..XX....XX.......X....X] duration:
30.002402663s trace:
https://jaeger.dev.storj.io/trace/75bbd55714594abb
FAIL (2025-12-29 15:55:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: S7CYPEJHQFCO2LZZS6ZIA6CH32WMJ7AMVJLREBKZO6IMRN2CVCDQ) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: FEIKCTMJJCRG22CS76TOQPX3FMHJ4OLPTS3MYAFJSGSY54S46XNA) context deadline exceeded
======LOGS======
Dec 29 15:55:29.640180914 (576h26m29.511263949s) [upload=505337]: Starting upload
Dec 29 15:55:29.640266207 (576h26m29.511349252s) [upload=505337]: Done starting upload
Dec 29 15:55:29.640406164 (576h26m29.511489209s) [upload=505337]: Uploading segments...
Dec 29 15:55:29.640519690 (576h26m29.511602735s) [upload=505337]: Got next segment. Inline: false
Dec 29 15:55:29.640571137 (576h26m29.511654172s) [upload=505337]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:55:29.640578492 (576h26m29.511661527s) [upload=505337]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:55:29.889648322 (576h26m29.760731367s) [seg_pos={0 0}, upload=505337]: Begin upload segment...
Dec 29 15:55:29.889701363 (576h26m29.760784408s) [seg_pos={0 0}, upload=505337]: Done begin upload segment.
Dec 29 15:55:29.889731862 (576h26m29.760814907s) [noise=true , address=145.79.227.45:4229 , piece_id=MIPS27E3X5NLANE74YFEZO3WKZUZJVHOTSEPGG4A5I6UFBXH5SAA, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.889829578 (576h26m29.760912613s) [noise=true , address=45.91.116.5:4298 , piece_id=BZPEIKV3K7LQ2KZ7MZZHCMSLEQM3UQFBB4OH2WXL6J3AF7LA3CKA, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.890278001 (576h26m29.761361046s) [noise=true , address=145.79.227.13:4804 , piece_id=PJWSRBMF3E5LKULZPROZ4EUJETAXHJBU2YTR2SDUM75GGJWEDQTQ, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.890606848 (576h26m29.761689883s) [noise=true , address=45.91.116.4:4939 , piece_id=CIEFNVNHT7LGC5SAF2Z5EPTXWZFROXWD3GAMPNGRS3AHWA3IUTQQ, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.890882402 (576h26m29.761965437s) [noise=true , address=145.79.227.5:4221 , piece_id=I45HGGEEZXUXAXLXAPWD6LQBW2P2X3JHYIEDUXHUPXVWNOW7BQUQ, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.891140964 (576h26m29.762223999s) [noise=true , address=45.91.116.6:4975 , piece_id=HSBQ4SUC5FHXUGITJYFLWO7HBZRYMXQNPQUZQH2I5GS5TUBFDVNA, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.891501762 (576h26m29.762584797s) [noise=true , address=145.79.227.12:4765 , piece_id=X3KOZ2UNK5Y2WLUXHEP5W6RBYXHHIISETW5H7RA25APPA75J377A, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.891829666 (576h26m29.762912701s) [noise=true , address=145.79.227.19:4078 , piece_id=4UNDCN5P72AOBJ5XFROMZDAM2GQMLK7GX5LQQAQ42L2B2TRE36GQ, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.892207926 (576h26m29.763290961s) [noise=true , address=145.79.227.47:4265 , piece_id=XV2PUO5DYJZ5IM6NMYRZF7RKZTEEIPKUMMP3PRQ4JFGFQMCYTM3A, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.894028833 (576h26m29.765111878s) [noise=true , address=145.79.227.39:4013 , piece_id=2ODPDKVUOIYOTUUVGM7SRXBFKF6CYC6GOAI24U7E5CLH35ILQRGQ, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.895792131 (576h26m29.766875176s) [noise=true , address=145.79.227.18:4006 , piece_id=HWDTN7APQXFULHEQJ6M3FWIX6KEZYGJVKGCRDI6CYMJ6F4MZF3QQ, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.897572311 (576h26m29.768655346s) [noise=true , address=45.91.116.14:4192 , piece_id=S7CYPEJHQFCO2LZZS6ZIA6CH32WMJ7AMVJLREBKZO6IMRN2CVCDQ, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.898843241 (576h26m29.769926287s) [noise=true , address=145.79.227.49:4337 , piece_id=CJO6AX7KMJJBAGIO5NJ63Z4LUJYH4DMNWMFCLTFAFUMLQDJHC5YA, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.900004663 (576h26m29.771087698s) [noise=true , address=45.91.116.11:4119 , piece_id=ZDXKSDGVQPOZX5SS7ANND6XAQJXZFF7UBCWFTUPGCHNSFHIEW4KA, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.901217452 (576h26m29.772300487s) [noise=true , address=145.79.227.6:4257 , piece_id=4WHKFOSFQSBKMDDCOWU45KJKKDDPYFBFVWWBNJASD44CLGAAXOYQ, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:29.902264686 (576h26m29.773347721s) [noise=true , address=77.90.31.14:4661 , piece_id=FEIKCTMJJCRG22CS76TOQPX3FMHJ4OLPTS3MYAFJSGSY54S46XNA, seg_pos={0 0}, upload=505337]: Uploading piece...
Dec 29 15:55:31.592207896 (576h26m31.463290941s) [upload=505337]: Next returned nil segment
Dec 29 15:55:31.592224318 (576h26m31.463307353s) [upload=505337]: Waiting for error group managing segments...
Dec 29 15:55:31.873758185 (576h26m31.74484122s) [noise=true , address=145.79.227.45:4229 , piece_id=MIPS27E3X5NLANE74YFEZO3WKZUZJVHOTSEPGG4A5I6UFBXH5SAA, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:31.875984494 (576h26m31.747067529s) [noise=true , address=145.79.227.5:4221 , piece_id=I45HGGEEZXUXAXLXAPWD6LQBW2P2X3JHYIEDUXHUPXVWNOW7BQUQ, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:31.876707992 (576h26m31.747791037s) [noise=true , address=145.79.227.6:4257 , piece_id=4WHKFOSFQSBKMDDCOWU45KJKKDDPYFBFVWWBNJASD44CLGAAXOYQ, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:31.889903798 (576h26m31.760986833s) [noise=true , address=145.79.227.18:4006 , piece_id=HWDTN7APQXFULHEQJ6M3FWIX6KEZYGJVKGCRDI6CYMJ6F4MZF3QQ, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:31.890572651 (576h26m31.761655686s) [noise=true , address=145.79.227.19:4078 , piece_id=4UNDCN5P72AOBJ5XFROMZDAM2GQMLK7GX5LQQAQ42L2B2TRE36GQ, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:31.895589305 (576h26m31.76667235s) [noise=true , address=145.79.227.13:4804 , piece_id=PJWSRBMF3E5LKULZPROZ4EUJETAXHJBU2YTR2SDUM75GGJWEDQTQ, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:31.900771702 (576h26m31.771854747s) [noise=true , address=145.79.227.12:4765 , piece_id=X3KOZ2UNK5Y2WLUXHEP5W6RBYXHHIISETW5H7RA25APPA75J377A, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:31.903510758 (576h26m31.774593813s) [noise=true , address=145.79.227.47:4265 , piece_id=XV2PUO5DYJZ5IM6NMYRZF7RKZTEEIPKUMMP3PRQ4JFGFQMCYTM3A, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:31.909782751 (576h26m31.780865796s) [noise=true , address=145.79.227.39:4013 , piece_id=2ODPDKVUOIYOTUUVGM7SRXBFKF6CYC6GOAI24U7E5CLH35ILQRGQ, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:32.167331647 (576h26m32.038414692s) [noise=true , address=45.91.116.5:4298 , piece_id=BZPEIKV3K7LQ2KZ7MZZHCMSLEQM3UQFBB4OH2WXL6J3AF7LA3CKA, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:32.618463797 (576h26m32.489546842s) [noise=true , address=45.91.116.11:4119 , piece_id=ZDXKSDGVQPOZX5SS7ANND6XAQJXZFF7UBCWFTUPGCHNSFHIEW4KA, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:32.854132278 (576h26m32.725215323s) [noise=true , address=77.90.31.14:4661 , piece_id=FEIKCTMJJCRG22CS76TOQPX3FMHJ4OLPTS3MYAFJSGSY54S46XNA, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:32.856202470 (576h26m32.727285505s) [noise=true , address=45.91.116.14:4192 , piece_id=S7CYPEJHQFCO2LZZS6ZIA6CH32WMJ7AMVJLREBKZO6IMRN2CVCDQ, seg_pos={0 0}, upload=505337]: Done uploading piece. err: <nil>
Dec 29 15:55:32.856220114 (576h26m32.727303149s) [seg_pos={0 0}, upload=505337]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:55:32.856560972 (576h26m32.727644017s) [noise=true , address=45.91.116.6:4975 , piece_id=HSBQ4SUC5FHXUGITJYFLWO7HBZRYMXQNPQUZQH2I5GS5TUBFDVNA, seg_pos={0 0}, upload=505337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 29 15:55:32.856591049 (576h26m32.727674094s) [noise=true , address=45.91.116.6:4975 , piece_id=HSBQ4SUC5FHXUGITJYFLWO7HBZRYMXQNPQUZQH2I5GS5TUBFDVNA, seg_pos={0 0}, upload=505337]: Long tail cancellation due to optimal threshold
Dec 29 15:55:32.856630064 (576h26m32.727713109s) [noise=true , address=45.91.116.4:4939 , piece_id=CIEFNVNHT7LGC5SAF2Z5EPTXWZFROXWD3GAMPNGRS3AHWA3IUTQQ, seg_pos={0 0}, upload=505337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39800->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 29 15:55:32.856639723 (576h26m32.727722768s) [noise=true , address=45.91.116.4:4939 , piece_id=CIEFNVNHT7LGC5SAF2Z5EPTXWZFROXWD3GAMPNGRS3AHWA3IUTQQ, seg_pos={0 0}, upload=505337]: Long tail cancellation due to optimal threshold
Dec 29 15:55:32.856756956 (576h26m32.727839991s) [noise=true , address=145.79.227.49:4337 , piece_id=CJO6AX7KMJJBAGIO5NJ63Z4LUJYH4DMNWMFCLTFAFUMLQDJHC5YA, seg_pos={0 0}, upload=505337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF
Dec 29 15:55:32.856764610 (576h26m32.727847645s) [noise=true , address=145.79.227.49:4337 , piece_id=CJO6AX7KMJJBAGIO5NJ63Z4LUJYH4DMNWMFCLTFAFUMLQDJHC5YA, seg_pos={0 0}, upload=505337]: Long tail cancellation due to optimal threshold
Dec 29 15:55:32.856784769 (576h26m32.727867804s) [seg_pos={0 0}, upload=505337]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:55:32.856894067 (576h26m32.727977112s) [upload=505337]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:55:32.856911990 (576h26m32.727995035s) [upload=505337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:55:33.153684143 (576h26m33.024767178s) [upload=505337]: Done uploading segments...
Dec 29 15:55:33.153705995 (576h26m33.02478903s) [upload=505337]: Upload finished. err: <nil>
Duration: 30.002423983s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X.........X..........................XX...................] duration:
43.714898143s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 15:55:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 43.714913221s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [X.XXX..X.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XXXXXXXXXXXXXX] duration:
5.002124307s trace:
https://jaeger.dev.storj.io/trace/24c765efc620c96f
FAIL (2025-12-29 15:55:28 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00214665s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [XXX.XXX..X...XXX..XX.X..X..XX.XX...X.X..XX....XX.......X....] duration:
6.25786351s trace:
https://jaeger.dev.storj.io/trace/41e69a21e7df56df
FAIL (2025-12-29 15:55:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: UYNEJM5O46TAC6D4QQRISIWWQAPVX2LKWD5C6MR5IMCYCIHCTCLQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: 2RXJROPWBPYOJB4XNNIVDCDUYUOKQONS77FKWLWGL3QLMGJ4SCLA) context deadline exceeded
======LOGS======
ing segments...
Dec 29 15:52:33.113308645 (576h23m32.98439171s) [upload=505297]: Got next segment. Inline: false
Dec 29 15:52:33.113353531 (576h23m32.984436576s) [upload=505297]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:52:33.113359291 (576h23m32.984442346s) [upload=505297]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:52:33.356574745 (576h23m33.22765778s) [seg_pos={0 0}, upload=505297]: Begin upload segment...
Dec 29 15:52:33.356623657 (576h23m33.227706693s) [seg_pos={0 0}, upload=505297]: Done begin upload segment.
Dec 29 15:52:33.356670446 (576h23m33.227753491s) [noise=true , address=145.79.227.12:4765 , piece_id=GLPY57LL2VW5A6XJGNF5MSC2YVLYEWFOO3ADOXQIB4THEI3J7EFQ, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.357312349 (576h23m33.228395384s) [noise=true , address=77.90.31.10:4520 , piece_id=VNFNZZWIGXOSCAQXQ6BMXY3CFS6PGE35UPBZYKB56IRJ3RS2ZLUQ, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.357783265 (576h23m33.2288663s) [noise=true , address=145.79.227.47:4266 , piece_id=C6RV4UZS3CDOPNTJSBZGD4WWM53V7T27EAJ4IPRSCQ6KI2QBXTSA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.358259242 (576h23m33.229342277s) [noise=true , address=77.90.31.21:4877 , piece_id=UYNEJM5O46TAC6D4QQRISIWWQAPVX2LKWD5C6MR5IMCYCIHCTCLQ, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.358696595 (576h23m33.22977964s) [noise=true , address=77.90.31.14:4663 , piece_id=2RXJROPWBPYOJB4XNNIVDCDUYUOKQONS77FKWLWGL3QLMGJ4SCLA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.359130992 (576h23m33.230214027s) [noise=true , address=145.79.227.48:4301 , piece_id=LLJIWWMZDZDJCODY7YJVQY27YQL4A36UBYEKGV47KJUREZTWFSCA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.359562725 (576h23m33.23064576s) [noise=true , address=145.79.227.45:4230 , piece_id=KIYGXG5H35W4O47M3HBSI4KJQ3SYSD6NNJIAYLKBTMQCVEUIBLZA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.361070435 (576h23m33.23215348s) [noise=true , address=145.79.227.33:4869 , piece_id=ID74MO4L7NZYFHMUWWRBIHSR5WJYK2SKWMGHBTIGFBKRWBBNWFBA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.365131138 (576h23m33.236214193s) [noise=true , address=45.91.116.10:4081 , piece_id=3PU7WITYZNSEFDSETDJYOSCRYDBPRTWO5FOQEQWCLYZGX3DBYL6Q, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.367723854 (576h23m33.238806889s) [noise=true , address=145.79.227.19:4078 , piece_id=CCYB7LPX6DJHKMRFVSM2D3UZWZH72WM4SF7XLJGPIXXC3RDD5POA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.368542724 (576h23m33.239625759s) [noise=true , address=45.91.116.8:4047 , piece_id=7O57X2KVXEJ6SHF7Y6NC6B24ADJ45ZF5AA35QI4WXRSC3GS2YH4A, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.371024630 (576h23m33.242107665s) [noise=true , address=145.79.227.9:4365 , piece_id=DZP53KNQIZ63XIJ7T65SCLBCQ2QJANM5MAX7AN54NR2YT2GY4U6A, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.372466505 (576h23m33.24354954s) [noise=true , address=145.79.227.51:4411 , piece_id=PODPAJ2IEUQA6TJRC7FHOWNKWNWMDPIPG3QXDI3IY7B2IWVECDPQ, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.373865559 (576h23m33.244948614s) [noise=true , address=145.79.227.50:4375 , piece_id=CTVQDPWDEPEOOSWL57JJ6CNIEKZOOAUOIJFLMNL2VPNV2VQ3MN3Q, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.374480460 (576h23m33.245563495s) [noise=true , address=45.91.116.11:4117 , piece_id=KHXD5HV5DUM2SECDMGZWJAUOQHDSBY2EPI2UTEGNJ57GVMMEUQNA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.375504771 (576h23m33.246587816s) [noise=true , address=145.79.227.11:4731 , piece_id=ZKTIGTQ4OJG3ZOV5R242FYWN23GJNMJOJQUSTKW2COX3WH2JGUEQ, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:35.074628983 (576h23m34.945712028s) [upload=505297]: Next returned nil segment
Dec 29 15:52:35.074720957 (576h23m34.945804022s) [upload=505297]: Waiting for error group managing segments...
Dec 29 15:52:35.358007782 (576h23m35.229090817s) [noise=true , address=145.79.227.51:4411 , piece_id=PODPAJ2IEUQA6TJRC7FHOWNKWNWMDPIPG3QXDI3IY7B2IWVECDPQ, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.358063257 (576h23m35.229146292s) [noise=true , address=145.79.227.45:4230 , piece_id=KIYGXG5H35W4O47M3HBSI4KJQ3SYSD6NNJIAYLKBTMQCVEUIBLZA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.364004791 (576h23m35.235087836s) [noise=true , address=145.79.227.19:4078 , piece_id=CCYB7LPX6DJHKMRFVSM2D3UZWZH72WM4SF7XLJGPIXXC3RDD5POA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.365266804 (576h23m35.236349849s) [noise=true , address=145.79.227.50:4375 , piece_id=CTVQDPWDEPEOOSWL57JJ6CNIEKZOOAUOIJFLMNL2VPNV2VQ3MN3Q, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.369359468 (576h23m35.240442503s) [noise=true , address=145.79.227.48:4301 , piece_id=LLJIWWMZDZDJCODY7YJVQY27YQL4A36UBYEKGV47KJUREZTWFSCA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.372684949 (576h23m35.243767984s) [noise=true , address=145.79.227.11:4731 , piece_id=ZKTIGTQ4OJG3ZOV5R242FYWN23GJNMJOJQUSTKW2COX3WH2JGUEQ, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.376723029 (576h23m35.247806064s) [noise=true , address=145.79.227.9:4365 , piece_id=DZP53KNQIZ63XIJ7T65SCLBCQ2QJANM5MAX7AN54NR2YT2GY4U6A, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.378716946 (576h23m35.249799981s) [noise=true , address=145.79.227.12:4765 , piece_id=GLPY57LL2VW5A6XJGNF5MSC2YVLYEWFOO3ADOXQIB4THEI3J7EFQ, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.387105650 (576h23m35.258188705s) [noise=true , address=145.79.227.47:4266 , piece_id=C6RV4UZS3CDOPNTJSBZGD4WWM53V7T27EAJ4IPRSCQ6KI2QBXTSA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.536904311 (576h23m35.407987356s) [noise=true , address=77.90.31.14:4663 , piece_id=2RXJROPWBPYOJB4XNNIVDCDUYUOKQONS77FKWLWGL3QLMGJ4SCLA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.581947891 (576h23m35.453030926s) [noise=true , address=77.90.31.21:4877 , piece_id=UYNEJM5O46TAC6D4QQRISIWWQAPVX2LKWD5C6MR5IMCYCIHCTCLQ, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.620922546 (576h23m35.492005591s) [noise=true , address=45.91.116.10:4081 , piece_id=3PU7WITYZNSEFDSETDJYOSCRYDBPRTWO5FOQEQWCLYZGX3DBYL6Q, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.772505715 (576h23m35.64358876s) [noise=true , address=45.91.116.8:4047 , piece_id=7O57X2KVXEJ6SHF7Y6NC6B24ADJ45ZF5AA35QI4WXRSC3GS2YH4A, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.772528959 (576h23m35.643611994s) [seg_pos={0 0}, upload=505297]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:52:35.772730042 (576h23m35.643813087s) [noise=true , address=45.91.116.11:4117 , piece_id=KHXD5HV5DUM2SECDMGZWJAUOQHDSBY2EPI2UTEGNJ57GVMMEUQNA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:48726->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:48726->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48726->45.91.116.11:4117: use of closed network connection
Dec 29 15:52:35.772738618 (576h23m35.643821653s) [noise=true , address=45.91.116.11:4117 , piece_id=KHXD5HV5DUM2SECDMGZWJAUOQHDSBY2EPI2UTEGNJ57GVMMEUQNA, seg_pos={0 0}, upload=505297]: Long tail cancellation due to optimal threshold
Dec 29 15:52:35.772748797 (576h23m35.643831843s) [noise=true , address=77.90.31.10:4520 , piece_id=VNFNZZWIGXOSCAQXQ6BMXY3CFS6PGE35UPBZYKB56IRJ3RS2ZLUQ, seg_pos={0 0}, upload=505297]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:53796->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:53796->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53796->77.90.31.10:4520: use of closed network connection
Dec 29 15:52:35.772759357 (576h23m35.643842402s) [noise=true , address=77.90.31.10:4520 , piece_id=VNFNZZWIGXOSCAQXQ6BMXY3CFS6PGE35UPBZYKB56IRJ3RS2ZLUQ, seg_pos={0 0}, upload=505297]: Long tail cancellation due to optimal threshold
Dec 29 15:52:35.772907189 (576h23m35.643990235s) [noise=true , address=145.79.227.33:4869 , piece_id=ID74MO4L7NZYFHMUWWRBIHSR5WJYK2SKWMGHBTIGFBKRWBBNWFBA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 29 15:52:35.772914934 (576h23m35.643997979s) [noise=true , address=145.79.227.33:4869 , piece_id=ID74MO4L7NZYFHMUWWRBIHSR5WJYK2SKWMGHBTIGFBKRWBBNWFBA, seg_pos={0 0}, upload=505297]: Long tail cancellation due to optimal threshold
Dec 29 15:52:35.773065280 (576h23m35.644148325s) [seg_pos={0 0}, upload=505297]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:52:35.773149411 (576h23m35.644232456s) [upload=505297]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:52:35.773155192 (576h23m35.644238227s) [upload=505297]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:52:36.067027903 (576h23m35.938110958s) [upload=505297]: Done uploading segments...
Dec 29 15:52:36.067053292 (576h23m35.938136347s) [upload=505297]: Upload finished. err: <nil>
Duration: 6.257891071s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........X................X...............................] duration:
17.10364937s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 15:55:06 UTC)
orbiter warning: context deadline exceeded
Duration: 17.103684186s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X.........X..........................XX..................] duration:
47.91977084s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 15:54:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 47.919798142s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [.XXX.XXX..X...XXX..XX.X..X..XX.XX...X.X..XX....XX.......X...] duration:
5.656491661s trace:
https://jaeger.dev.storj.io/trace/41e69a21e7df56df
FAIL (2025-12-29 15:54:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: UYNEJM5O46TAC6D4QQRISIWWQAPVX2LKWD5C6MR5IMCYCIHCTCLQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: 2RXJROPWBPYOJB4XNNIVDCDUYUOKQONS77FKWLWGL3QLMGJ4SCLA) context deadline exceeded
======LOGS======
ing segments...
Dec 29 15:52:33.113308645 (576h23m32.98439171s) [upload=505297]: Got next segment. Inline: false
Dec 29 15:52:33.113353531 (576h23m32.984436576s) [upload=505297]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:52:33.113359291 (576h23m32.984442346s) [upload=505297]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:52:33.356574745 (576h23m33.22765778s) [seg_pos={0 0}, upload=505297]: Begin upload segment...
Dec 29 15:52:33.356623657 (576h23m33.227706693s) [seg_pos={0 0}, upload=505297]: Done begin upload segment.
Dec 29 15:52:33.356670446 (576h23m33.227753491s) [noise=true , address=145.79.227.12:4765 , piece_id=GLPY57LL2VW5A6XJGNF5MSC2YVLYEWFOO3ADOXQIB4THEI3J7EFQ, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.357312349 (576h23m33.228395384s) [noise=true , address=77.90.31.10:4520 , piece_id=VNFNZZWIGXOSCAQXQ6BMXY3CFS6PGE35UPBZYKB56IRJ3RS2ZLUQ, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.357783265 (576h23m33.2288663s) [noise=true , address=145.79.227.47:4266 , piece_id=C6RV4UZS3CDOPNTJSBZGD4WWM53V7T27EAJ4IPRSCQ6KI2QBXTSA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.358259242 (576h23m33.229342277s) [noise=true , address=77.90.31.21:4877 , piece_id=UYNEJM5O46TAC6D4QQRISIWWQAPVX2LKWD5C6MR5IMCYCIHCTCLQ, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.358696595 (576h23m33.22977964s) [noise=true , address=77.90.31.14:4663 , piece_id=2RXJROPWBPYOJB4XNNIVDCDUYUOKQONS77FKWLWGL3QLMGJ4SCLA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.359130992 (576h23m33.230214027s) [noise=true , address=145.79.227.48:4301 , piece_id=LLJIWWMZDZDJCODY7YJVQY27YQL4A36UBYEKGV47KJUREZTWFSCA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.359562725 (576h23m33.23064576s) [noise=true , address=145.79.227.45:4230 , piece_id=KIYGXG5H35W4O47M3HBSI4KJQ3SYSD6NNJIAYLKBTMQCVEUIBLZA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.361070435 (576h23m33.23215348s) [noise=true , address=145.79.227.33:4869 , piece_id=ID74MO4L7NZYFHMUWWRBIHSR5WJYK2SKWMGHBTIGFBKRWBBNWFBA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.365131138 (576h23m33.236214193s) [noise=true , address=45.91.116.10:4081 , piece_id=3PU7WITYZNSEFDSETDJYOSCRYDBPRTWO5FOQEQWCLYZGX3DBYL6Q, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.367723854 (576h23m33.238806889s) [noise=true , address=145.79.227.19:4078 , piece_id=CCYB7LPX6DJHKMRFVSM2D3UZWZH72WM4SF7XLJGPIXXC3RDD5POA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.368542724 (576h23m33.239625759s) [noise=true , address=45.91.116.8:4047 , piece_id=7O57X2KVXEJ6SHF7Y6NC6B24ADJ45ZF5AA35QI4WXRSC3GS2YH4A, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.371024630 (576h23m33.242107665s) [noise=true , address=145.79.227.9:4365 , piece_id=DZP53KNQIZ63XIJ7T65SCLBCQ2QJANM5MAX7AN54NR2YT2GY4U6A, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.372466505 (576h23m33.24354954s) [noise=true , address=145.79.227.51:4411 , piece_id=PODPAJ2IEUQA6TJRC7FHOWNKWNWMDPIPG3QXDI3IY7B2IWVECDPQ, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.373865559 (576h23m33.244948614s) [noise=true , address=145.79.227.50:4375 , piece_id=CTVQDPWDEPEOOSWL57JJ6CNIEKZOOAUOIJFLMNL2VPNV2VQ3MN3Q, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.374480460 (576h23m33.245563495s) [noise=true , address=45.91.116.11:4117 , piece_id=KHXD5HV5DUM2SECDMGZWJAUOQHDSBY2EPI2UTEGNJ57GVMMEUQNA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.375504771 (576h23m33.246587816s) [noise=true , address=145.79.227.11:4731 , piece_id=ZKTIGTQ4OJG3ZOV5R242FYWN23GJNMJOJQUSTKW2COX3WH2JGUEQ, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:35.074628983 (576h23m34.945712028s) [upload=505297]: Next returned nil segment
Dec 29 15:52:35.074720957 (576h23m34.945804022s) [upload=505297]: Waiting for error group managing segments...
Dec 29 15:52:35.358007782 (576h23m35.229090817s) [noise=true , address=145.79.227.51:4411 , piece_id=PODPAJ2IEUQA6TJRC7FHOWNKWNWMDPIPG3QXDI3IY7B2IWVECDPQ, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.358063257 (576h23m35.229146292s) [noise=true , address=145.79.227.45:4230 , piece_id=KIYGXG5H35W4O47M3HBSI4KJQ3SYSD6NNJIAYLKBTMQCVEUIBLZA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.364004791 (576h23m35.235087836s) [noise=true , address=145.79.227.19:4078 , piece_id=CCYB7LPX6DJHKMRFVSM2D3UZWZH72WM4SF7XLJGPIXXC3RDD5POA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.365266804 (576h23m35.236349849s) [noise=true , address=145.79.227.50:4375 , piece_id=CTVQDPWDEPEOOSWL57JJ6CNIEKZOOAUOIJFLMNL2VPNV2VQ3MN3Q, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.369359468 (576h23m35.240442503s) [noise=true , address=145.79.227.48:4301 , piece_id=LLJIWWMZDZDJCODY7YJVQY27YQL4A36UBYEKGV47KJUREZTWFSCA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.372684949 (576h23m35.243767984s) [noise=true , address=145.79.227.11:4731 , piece_id=ZKTIGTQ4OJG3ZOV5R242FYWN23GJNMJOJQUSTKW2COX3WH2JGUEQ, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.376723029 (576h23m35.247806064s) [noise=true , address=145.79.227.9:4365 , piece_id=DZP53KNQIZ63XIJ7T65SCLBCQ2QJANM5MAX7AN54NR2YT2GY4U6A, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.378716946 (576h23m35.249799981s) [noise=true , address=145.79.227.12:4765 , piece_id=GLPY57LL2VW5A6XJGNF5MSC2YVLYEWFOO3ADOXQIB4THEI3J7EFQ, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.387105650 (576h23m35.258188705s) [noise=true , address=145.79.227.47:4266 , piece_id=C6RV4UZS3CDOPNTJSBZGD4WWM53V7T27EAJ4IPRSCQ6KI2QBXTSA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.536904311 (576h23m35.407987356s) [noise=true , address=77.90.31.14:4663 , piece_id=2RXJROPWBPYOJB4XNNIVDCDUYUOKQONS77FKWLWGL3QLMGJ4SCLA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.581947891 (576h23m35.453030926s) [noise=true , address=77.90.31.21:4877 , piece_id=UYNEJM5O46TAC6D4QQRISIWWQAPVX2LKWD5C6MR5IMCYCIHCTCLQ, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.620922546 (576h23m35.492005591s) [noise=true , address=45.91.116.10:4081 , piece_id=3PU7WITYZNSEFDSETDJYOSCRYDBPRTWO5FOQEQWCLYZGX3DBYL6Q, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.772505715 (576h23m35.64358876s) [noise=true , address=45.91.116.8:4047 , piece_id=7O57X2KVXEJ6SHF7Y6NC6B24ADJ45ZF5AA35QI4WXRSC3GS2YH4A, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.772528959 (576h23m35.643611994s) [seg_pos={0 0}, upload=505297]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:52:35.772730042 (576h23m35.643813087s) [noise=true , address=45.91.116.11:4117 , piece_id=KHXD5HV5DUM2SECDMGZWJAUOQHDSBY2EPI2UTEGNJ57GVMMEUQNA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:48726->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:48726->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48726->45.91.116.11:4117: use of closed network connection
Dec 29 15:52:35.772738618 (576h23m35.643821653s) [noise=true , address=45.91.116.11:4117 , piece_id=KHXD5HV5DUM2SECDMGZWJAUOQHDSBY2EPI2UTEGNJ57GVMMEUQNA, seg_pos={0 0}, upload=505297]: Long tail cancellation due to optimal threshold
Dec 29 15:52:35.772748797 (576h23m35.643831843s) [noise=true , address=77.90.31.10:4520 , piece_id=VNFNZZWIGXOSCAQXQ6BMXY3CFS6PGE35UPBZYKB56IRJ3RS2ZLUQ, seg_pos={0 0}, upload=505297]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:53796->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:53796->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53796->77.90.31.10:4520: use of closed network connection
Dec 29 15:52:35.772759357 (576h23m35.643842402s) [noise=true , address=77.90.31.10:4520 , piece_id=VNFNZZWIGXOSCAQXQ6BMXY3CFS6PGE35UPBZYKB56IRJ3RS2ZLUQ, seg_pos={0 0}, upload=505297]: Long tail cancellation due to optimal threshold
Dec 29 15:52:35.772907189 (576h23m35.643990235s) [noise=true , address=145.79.227.33:4869 , piece_id=ID74MO4L7NZYFHMUWWRBIHSR5WJYK2SKWMGHBTIGFBKRWBBNWFBA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 29 15:52:35.772914934 (576h23m35.643997979s) [noise=true , address=145.79.227.33:4869 , piece_id=ID74MO4L7NZYFHMUWWRBIHSR5WJYK2SKWMGHBTIGFBKRWBBNWFBA, seg_pos={0 0}, upload=505297]: Long tail cancellation due to optimal threshold
Dec 29 15:52:35.773065280 (576h23m35.644148325s) [seg_pos={0 0}, upload=505297]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:52:35.773149411 (576h23m35.644232456s) [upload=505297]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:52:35.773155192 (576h23m35.644238227s) [upload=505297]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:52:36.067027903 (576h23m35.938110958s) [upload=505297]: Done uploading segments...
Dec 29 15:52:36.067053292 (576h23m35.938136347s) [upload=505297]: Upload finished. err: <nil>
Duration: 5.656520566s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [XX.XXX..X.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XXXXXXXXXXXXX] duration:
5.001239081s trace:
https://jaeger.dev.storj.io/trace/680541d6e8d1c080
FAIL (2025-12-29 15:54:20 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001251895s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............X................X..............................] duration:
9.062819458s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 15:54:17 UTC)
orbiter warning: context deadline exceeded
Duration: 9.062852251s
edge
gateway
us-select-1
|