Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..............................X........X...........X..X.....] duration:
11.768955234s trace:
https://jaeger.dev.storj.io/trace/17a3dfac1fc2c808
FAIL (2025-12-30 13:02:17 UTC)
orbiter warning: context deadline exceeded
Duration: 11.768970113s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.................X.......................X....X.............] duration:
12.01991568s trace:
https://jaeger.dev.storj.io/trace/7b47a373389cd7c3
FAIL (2025-12-30 13:01:51 UTC)
orbiter warning: context deadline exceeded
Duration: 12.019945136s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [XX..X.......X.XX..X..X.XX...............X..X.X..X....X......] duration:
5.657716735s trace:
https://jaeger.dev.storj.io/trace/23ad8207da0148f9
FAIL (2025-12-30 13:01:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ) context deadline exceeded
======LOGS======
ments...
Dec 30 12:55:52.163563927 (597h26m52.034646982s) [upload=523705]: Got next segment. Inline: false
Dec 30 12:55:52.163612819 (597h26m52.034695864s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 12:55:52.163618821 (597h26m52.034701866s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 12:55:52.410969244 (597h26m52.282052279s) [seg_pos={0 0}, upload=523705]: Begin upload segment...
Dec 30 12:55:52.411037163 (597h26m52.282120198s) [seg_pos={0 0}, upload=523705]: Done begin upload segment.
Dec 30 12:55:52.411066920 (597h26m52.282149955s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.411202477 (597h26m52.282285522s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.411721015 (597h26m52.28280405s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.412140103 (597h26m52.283223138s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.412521419 (597h26m52.283604454s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.412888228 (597h26m52.283971263s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.413237172 (597h26m52.284320207s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.413653144 (597h26m52.284736179s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.414092440 (597h26m52.285175475s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.415787007 (597h26m52.286870042s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.415936882 (597h26m52.287019917s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.416357675 (597h26m52.28744072s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.416794676 (597h26m52.287877721s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.417238962 (597h26m52.288321997s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.417715961 (597h26m52.288798996s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.418153584 (597h26m52.289236619s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:54.018730327 (597h26m53.889813362s) [upload=523705]: Next returned nil segment
Dec 30 12:55:54.018743952 (597h26m53.889826997s) [upload=523705]: Waiting for error group managing segments...
Dec 30 12:55:54.399128062 (597h26m54.270211097s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.399875084 (597h26m54.270958129s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.404232521 (597h26m54.275315556s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.424052321 (597h26m54.295135356s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.433077286 (597h26m54.304160321s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.437149530 (597h26m54.308232575s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.480458237 (597h26m54.351541282s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.689140229 (597h26m54.560223274s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.711163926 (597h26m54.582246961s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.730262152 (597h26m54.601345197s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.877886751 (597h26m54.748969796s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.889062301 (597h26m54.760145336s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:55.074468032 (597h26m54.945551077s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:55.074489473 (597h26m54.945572518s) [seg_pos={0 0}, upload=523705]: Segment reached optimal threshold of 13 pieces.
Dec 30 12:55:55.074727766 (597h26m54.945810811s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection
Dec 30 12:55:55.074735361 (597h26m54.945818396s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold
Dec 30 12:55:55.074903922 (597h26m54.945986967s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection
Dec 30 12:55:55.074929440 (597h26m54.946012485s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold
Dec 30 12:55:55.074969206 (597h26m54.946052241s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 30 12:55:55.074975178 (597h26m54.946058213s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold
Dec 30 12:55:55.075042837 (597h26m54.946125872s) [seg_pos={0 0}, upload=523705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 12:55:55.075106016 (597h26m54.946189051s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 12:55:55.075110695 (597h26m54.94619373s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 12:55:55.365579192 (597h26m55.236662227s) [upload=523705]: Done uploading segments...
Dec 30 12:55:55.365601594 (597h26m55.236684639s) [upload=523705]: Upload finished. err: <nil>
Duration: 5.65775631s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
7 out of 60 is failed (threshold=3) [............X.....X.X...X.........X....X........X...........] duration:
29.45699545s trace:
https://jaeger.dev.storj.io/trace/1e6433ec4a141391
FAIL (2025-12-30 13:01:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
[noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.352305080 (597h22m35.223388115s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.353083031 (597h22m35.224166066s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.353482140 (597h22m35.224565175s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.354315267 (597h22m35.225398302s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.354723675 (597h22m35.22580671s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.355352552 (597h22m35.226435587s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.356361994 (597h22m35.227445029s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.356578216 (597h22m35.227661261s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.357840370 (597h22m35.228923405s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.359039433 (597h22m35.230122468s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.359333493 (597h22m35.230416538s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.360694154 (597h22m35.231777189s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.361764952 (597h22m35.232847987s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.363365710 (597h22m35.234448755s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.363684648 (597h22m35.234767683s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:36.749009517 (597h22m36.620092552s) [upload=523642]: Next returned nil segment
Dec 30 12:51:36.749023393 (597h22m36.620106428s) [upload=523642]: Waiting for error group managing segments...
Dec 30 12:51:37.208596330 (597h22m37.079679376s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.240881687 (597h22m37.111964722s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.294293140 (597h22m37.165376175s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.302040142 (597h22m37.173123177s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.308919160 (597h22m37.180002196s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.370286944 (597h22m37.24136999s) [noise=true , address=77.90.31.10:4545 , piece_id=YQV3IPXMIBSO5ZCISFUBGRGHE6FIXLKTLNYKPWR42VNBKNOGPG5Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.397564325 (597h22m37.26864737s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.481372759 (597h22m37.352455794s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.868609147 (597h22m37.739692192s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.122711215 (597h22m37.993794251s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.156332275 (597h22m38.02741531s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.210830938 (597h22m38.081913983s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.318916249 (597h22m38.189999284s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.528107081 (597h22m38.399190127s) [noise=true , address=193.149.164.143:4694 , piece_id=AOZVLCCSBEYZX3WES7JJVOA2BECEBPII3A33PG5KCSQY6LSZ6Y3A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.528133793 (597h22m38.399216838s) [seg_pos={0 0}, upload=523642]: Segment reached optimal threshold of 14 pieces.
Dec 30 12:51:38.528364652 (597h22m38.399447687s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528373429 (597h22m38.399456464s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.528672969 (597h22m38.399756014s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528682567 (597h22m38.399765602s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.528839085 (597h22m38.39992212s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528847421 (597h22m38.399930476s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.529001655 (597h22m38.4000847s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection
Dec 30 12:51:38.529010362 (597h22m38.400093407s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.529029568 (597h22m38.400112603s) [seg_pos={0 0}, upload=523642]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 12:51:38.529107577 (597h22m38.400190612s) [upload=523642]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 12:51:38.529114079 (597h22m38.400197124s) [upload=523642]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 12:51:38.828356312 (597h22m38.699439387s) [upload=523642]: Done uploading segments...
Dec 30 12:51:38.828379166 (597h22m38.699462231s) [upload=523642]: Upload finished. err: <nil>
Duration: 29.457021408s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...............................X........X...........X..X....] duration:
15.700676517s trace:
https://jaeger.dev.storj.io/trace/17a3dfac1fc2c808
FAIL (2025-12-30 13:01:30 UTC)
orbiter warning: context deadline exceeded
Duration: 15.700701794s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.X] duration:
5.001084467s trace:
https://jaeger.dev.storj.io/trace/4a194d3d63581f75
FAIL (2025-12-30 13:01:19 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001110125s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [.XX..X.......X.XX..X..X.XX...............X..X.X..X....X.....] duration:
28.381322226s trace:
https://jaeger.dev.storj.io/trace/23ad8207da0148f9
FAIL (2025-12-30 13:01:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ) context deadline exceeded
======LOGS======
ments...
Dec 30 12:55:52.163563927 (597h26m52.034646982s) [upload=523705]: Got next segment. Inline: false
Dec 30 12:55:52.163612819 (597h26m52.034695864s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 12:55:52.163618821 (597h26m52.034701866s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 12:55:52.410969244 (597h26m52.282052279s) [seg_pos={0 0}, upload=523705]: Begin upload segment...
Dec 30 12:55:52.411037163 (597h26m52.282120198s) [seg_pos={0 0}, upload=523705]: Done begin upload segment.
Dec 30 12:55:52.411066920 (597h26m52.282149955s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.411202477 (597h26m52.282285522s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.411721015 (597h26m52.28280405s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.412140103 (597h26m52.283223138s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.412521419 (597h26m52.283604454s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.412888228 (597h26m52.283971263s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.413237172 (597h26m52.284320207s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.413653144 (597h26m52.284736179s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.414092440 (597h26m52.285175475s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.415787007 (597h26m52.286870042s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.415936882 (597h26m52.287019917s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.416357675 (597h26m52.28744072s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.416794676 (597h26m52.287877721s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.417238962 (597h26m52.288321997s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.417715961 (597h26m52.288798996s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.418153584 (597h26m52.289236619s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:54.018730327 (597h26m53.889813362s) [upload=523705]: Next returned nil segment
Dec 30 12:55:54.018743952 (597h26m53.889826997s) [upload=523705]: Waiting for error group managing segments...
Dec 30 12:55:54.399128062 (597h26m54.270211097s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.399875084 (597h26m54.270958129s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.404232521 (597h26m54.275315556s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.424052321 (597h26m54.295135356s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.433077286 (597h26m54.304160321s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.437149530 (597h26m54.308232575s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.480458237 (597h26m54.351541282s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.689140229 (597h26m54.560223274s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.711163926 (597h26m54.582246961s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.730262152 (597h26m54.601345197s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.877886751 (597h26m54.748969796s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.889062301 (597h26m54.760145336s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:55.074468032 (597h26m54.945551077s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:55.074489473 (597h26m54.945572518s) [seg_pos={0 0}, upload=523705]: Segment reached optimal threshold of 13 pieces.
Dec 30 12:55:55.074727766 (597h26m54.945810811s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection
Dec 30 12:55:55.074735361 (597h26m54.945818396s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold
Dec 30 12:55:55.074903922 (597h26m54.945986967s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection
Dec 30 12:55:55.074929440 (597h26m54.946012485s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold
Dec 30 12:55:55.074969206 (597h26m54.946052241s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 30 12:55:55.074975178 (597h26m54.946058213s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold
Dec 30 12:55:55.075042837 (597h26m54.946125872s) [seg_pos={0 0}, upload=523705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 12:55:55.075106016 (597h26m54.946189051s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 12:55:55.075110695 (597h26m54.94619373s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 12:55:55.365579192 (597h26m55.236662227s) [upload=523705]: Done uploading segments...
Dec 30 12:55:55.365601594 (597h26m55.236684639s) [upload=523705]: Upload finished. err: <nil>
Duration: 28.381347434s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..................X.......................X....X............] duration:
17.71507809s trace:
https://jaeger.dev.storj.io/trace/7b47a373389cd7c3
FAIL (2025-12-30 13:01:11 UTC)
orbiter warning: context deadline exceeded
Duration: 17.715090103s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [.XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.] duration:
3.830721569s trace:
https://jaeger.dev.storj.io/trace/7bfe418e178b51b1
FAIL (2025-12-30 13:00:58 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.830800348s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [................................X........X...........X..X...] duration:
19.822188341s trace:
https://jaeger.dev.storj.io/trace/17a3dfac1fc2c808
FAIL (2025-12-30 13:00:56 UTC)
orbiter warning: context deadline exceeded
Duration: 19.822205413s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...................X.......................X....X...........] duration:
17.486281869s trace:
https://jaeger.dev.storj.io/trace/7b47a373389cd7c3
FAIL (2025-12-30 13:00:46 UTC)
orbiter warning: context deadline exceeded
Duration: 17.48629873s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.................................X........X...........X..X..] duration:
23.059003725s trace:
https://jaeger.dev.storj.io/trace/17a3dfac1fc2c808
FAIL (2025-12-30 13:00:36 UTC)
orbiter warning: context deadline exceeded
Duration: 23.059035156s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
7 out of 60 is failed (threshold=3) [.............X.....X.X...X.........X....X........X..........] duration:
27.593743599s trace:
https://jaeger.dev.storj.io/trace/1e6433ec4a141391
FAIL (2025-12-30 13:00:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
[noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.352305080 (597h22m35.223388115s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.353083031 (597h22m35.224166066s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.353482140 (597h22m35.224565175s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.354315267 (597h22m35.225398302s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.354723675 (597h22m35.22580671s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.355352552 (597h22m35.226435587s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.356361994 (597h22m35.227445029s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.356578216 (597h22m35.227661261s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.357840370 (597h22m35.228923405s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.359039433 (597h22m35.230122468s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.359333493 (597h22m35.230416538s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.360694154 (597h22m35.231777189s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.361764952 (597h22m35.232847987s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.363365710 (597h22m35.234448755s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.363684648 (597h22m35.234767683s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:36.749009517 (597h22m36.620092552s) [upload=523642]: Next returned nil segment
Dec 30 12:51:36.749023393 (597h22m36.620106428s) [upload=523642]: Waiting for error group managing segments...
Dec 30 12:51:37.208596330 (597h22m37.079679376s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.240881687 (597h22m37.111964722s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.294293140 (597h22m37.165376175s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.302040142 (597h22m37.173123177s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.308919160 (597h22m37.180002196s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.370286944 (597h22m37.24136999s) [noise=true , address=77.90.31.10:4545 , piece_id=YQV3IPXMIBSO5ZCISFUBGRGHE6FIXLKTLNYKPWR42VNBKNOGPG5Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.397564325 (597h22m37.26864737s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.481372759 (597h22m37.352455794s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.868609147 (597h22m37.739692192s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.122711215 (597h22m37.993794251s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.156332275 (597h22m38.02741531s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.210830938 (597h22m38.081913983s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.318916249 (597h22m38.189999284s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.528107081 (597h22m38.399190127s) [noise=true , address=193.149.164.143:4694 , piece_id=AOZVLCCSBEYZX3WES7JJVOA2BECEBPII3A33PG5KCSQY6LSZ6Y3A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.528133793 (597h22m38.399216838s) [seg_pos={0 0}, upload=523642]: Segment reached optimal threshold of 14 pieces.
Dec 30 12:51:38.528364652 (597h22m38.399447687s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528373429 (597h22m38.399456464s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.528672969 (597h22m38.399756014s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528682567 (597h22m38.399765602s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.528839085 (597h22m38.39992212s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528847421 (597h22m38.399930476s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.529001655 (597h22m38.4000847s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection
Dec 30 12:51:38.529010362 (597h22m38.400093407s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.529029568 (597h22m38.400112603s) [seg_pos={0 0}, upload=523642]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 12:51:38.529107577 (597h22m38.400190612s) [upload=523642]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 12:51:38.529114079 (597h22m38.400197124s) [upload=523642]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 12:51:38.828356312 (597h22m38.699439387s) [upload=523642]: Done uploading segments...
Dec 30 12:51:38.828379166 (597h22m38.699462231s) [upload=523642]: Upload finished. err: <nil>
Duration: 27.593772425s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [....................X.......................X....X..........] duration:
8.699956652s trace:
https://jaeger.dev.storj.io/trace/7b47a373389cd7c3
FAIL (2025-12-30 13:00:17 UTC)
orbiter warning: context deadline exceeded
Duration: 8.699996527s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [..XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX] duration:
5.001350594s trace:
https://jaeger.dev.storj.io/trace/7bfe418e178b51b1
FAIL (2025-12-30 13:00:11 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001364349s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [..XX..X.......X.XX..X..X.XX...............X..X.X..X....X....] duration:
6.022754283s trace:
https://jaeger.dev.storj.io/trace/23ad8207da0148f9
FAIL (2025-12-30 12:59:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ) context deadline exceeded
======LOGS======
ments...
Dec 30 12:55:52.163563927 (597h26m52.034646982s) [upload=523705]: Got next segment. Inline: false
Dec 30 12:55:52.163612819 (597h26m52.034695864s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 12:55:52.163618821 (597h26m52.034701866s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 12:55:52.410969244 (597h26m52.282052279s) [seg_pos={0 0}, upload=523705]: Begin upload segment...
Dec 30 12:55:52.411037163 (597h26m52.282120198s) [seg_pos={0 0}, upload=523705]: Done begin upload segment.
Dec 30 12:55:52.411066920 (597h26m52.282149955s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.411202477 (597h26m52.282285522s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.411721015 (597h26m52.28280405s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.412140103 (597h26m52.283223138s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.412521419 (597h26m52.283604454s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.412888228 (597h26m52.283971263s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.413237172 (597h26m52.284320207s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.413653144 (597h26m52.284736179s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.414092440 (597h26m52.285175475s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.415787007 (597h26m52.286870042s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.415936882 (597h26m52.287019917s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.416357675 (597h26m52.28744072s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.416794676 (597h26m52.287877721s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.417238962 (597h26m52.288321997s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.417715961 (597h26m52.288798996s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.418153584 (597h26m52.289236619s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:54.018730327 (597h26m53.889813362s) [upload=523705]: Next returned nil segment
Dec 30 12:55:54.018743952 (597h26m53.889826997s) [upload=523705]: Waiting for error group managing segments...
Dec 30 12:55:54.399128062 (597h26m54.270211097s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.399875084 (597h26m54.270958129s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.404232521 (597h26m54.275315556s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.424052321 (597h26m54.295135356s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.433077286 (597h26m54.304160321s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.437149530 (597h26m54.308232575s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.480458237 (597h26m54.351541282s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.689140229 (597h26m54.560223274s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.711163926 (597h26m54.582246961s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.730262152 (597h26m54.601345197s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.877886751 (597h26m54.748969796s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.889062301 (597h26m54.760145336s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:55.074468032 (597h26m54.945551077s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:55.074489473 (597h26m54.945572518s) [seg_pos={0 0}, upload=523705]: Segment reached optimal threshold of 13 pieces.
Dec 30 12:55:55.074727766 (597h26m54.945810811s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection
Dec 30 12:55:55.074735361 (597h26m54.945818396s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold
Dec 30 12:55:55.074903922 (597h26m54.945986967s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection
Dec 30 12:55:55.074929440 (597h26m54.946012485s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold
Dec 30 12:55:55.074969206 (597h26m54.946052241s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 30 12:55:55.074975178 (597h26m54.946058213s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold
Dec 30 12:55:55.075042837 (597h26m54.946125872s) [seg_pos={0 0}, upload=523705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 12:55:55.075106016 (597h26m54.946189051s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 12:55:55.075110695 (597h26m54.94619373s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 12:55:55.365579192 (597h26m55.236662227s) [upload=523705]: Done uploading segments...
Dec 30 12:55:55.365601594 (597h26m55.236684639s) [upload=523705]: Upload finished. err: <nil>
Duration: 6.022780864s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
7 out of 60 is failed (threshold=3) [..............X.....X.X...X.........X....X........X.........] duration:
36.548519957s trace:
https://jaeger.dev.storj.io/trace/1e6433ec4a141391
FAIL (2025-12-30 12:59:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
[noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.352305080 (597h22m35.223388115s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.353083031 (597h22m35.224166066s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.353482140 (597h22m35.224565175s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.354315267 (597h22m35.225398302s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.354723675 (597h22m35.22580671s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.355352552 (597h22m35.226435587s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.356361994 (597h22m35.227445029s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.356578216 (597h22m35.227661261s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.357840370 (597h22m35.228923405s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.359039433 (597h22m35.230122468s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.359333493 (597h22m35.230416538s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.360694154 (597h22m35.231777189s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.361764952 (597h22m35.232847987s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.363365710 (597h22m35.234448755s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.363684648 (597h22m35.234767683s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:36.749009517 (597h22m36.620092552s) [upload=523642]: Next returned nil segment
Dec 30 12:51:36.749023393 (597h22m36.620106428s) [upload=523642]: Waiting for error group managing segments...
Dec 30 12:51:37.208596330 (597h22m37.079679376s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.240881687 (597h22m37.111964722s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.294293140 (597h22m37.165376175s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.302040142 (597h22m37.173123177s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.308919160 (597h22m37.180002196s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.370286944 (597h22m37.24136999s) [noise=true , address=77.90.31.10:4545 , piece_id=YQV3IPXMIBSO5ZCISFUBGRGHE6FIXLKTLNYKPWR42VNBKNOGPG5Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.397564325 (597h22m37.26864737s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.481372759 (597h22m37.352455794s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.868609147 (597h22m37.739692192s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.122711215 (597h22m37.993794251s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.156332275 (597h22m38.02741531s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.210830938 (597h22m38.081913983s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.318916249 (597h22m38.189999284s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.528107081 (597h22m38.399190127s) [noise=true , address=193.149.164.143:4694 , piece_id=AOZVLCCSBEYZX3WES7JJVOA2BECEBPII3A33PG5KCSQY6LSZ6Y3A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.528133793 (597h22m38.399216838s) [seg_pos={0 0}, upload=523642]: Segment reached optimal threshold of 14 pieces.
Dec 30 12:51:38.528364652 (597h22m38.399447687s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528373429 (597h22m38.399456464s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.528672969 (597h22m38.399756014s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528682567 (597h22m38.399765602s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.528839085 (597h22m38.39992212s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528847421 (597h22m38.399930476s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.529001655 (597h22m38.4000847s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection
Dec 30 12:51:38.529010362 (597h22m38.400093407s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.529029568 (597h22m38.400112603s) [seg_pos={0 0}, upload=523642]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 12:51:38.529107577 (597h22m38.400190612s) [upload=523642]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 12:51:38.529114079 (597h22m38.400197124s) [upload=523642]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 12:51:38.828356312 (597h22m38.699439387s) [upload=523642]: Done uploading segments...
Dec 30 12:51:38.828379166 (597h22m38.699462231s) [upload=523642]: Upload finished. err: <nil>
Duration: 36.548534545s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.....................X.......................X....X.........] duration:
19.337532924s trace:
https://jaeger.dev.storj.io/trace/7b47a373389cd7c3
FAIL (2025-12-30 12:59:29 UTC)
orbiter warning: context deadline exceeded
Duration: 19.337564793s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..................................X........X...........X..X.] duration:
16.851347789s trace:
https://jaeger.dev.storj.io/trace/17a3dfac1fc2c808
FAIL (2025-12-30 12:59:15 UTC)
orbiter warning: context deadline exceeded
Duration: 16.85137475s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [X..XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXX] duration:
5.001059251s trace:
https://jaeger.dev.storj.io/trace/5bbf8e2463bd5448
FAIL (2025-12-30 12:59:11 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001084139s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
7 out of 60 is failed (threshold=3) [...............X.....X.X...X.........X....X........X........] duration:
36.315730721s trace:
https://jaeger.dev.storj.io/trace/1e6433ec4a141391
FAIL (2025-12-30 12:58:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
[noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.352305080 (597h22m35.223388115s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.353083031 (597h22m35.224166066s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.353482140 (597h22m35.224565175s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.354315267 (597h22m35.225398302s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.354723675 (597h22m35.22580671s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.355352552 (597h22m35.226435587s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.356361994 (597h22m35.227445029s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.356578216 (597h22m35.227661261s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.357840370 (597h22m35.228923405s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.359039433 (597h22m35.230122468s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.359333493 (597h22m35.230416538s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.360694154 (597h22m35.231777189s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.361764952 (597h22m35.232847987s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.363365710 (597h22m35.234448755s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.363684648 (597h22m35.234767683s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:36.749009517 (597h22m36.620092552s) [upload=523642]: Next returned nil segment
Dec 30 12:51:36.749023393 (597h22m36.620106428s) [upload=523642]: Waiting for error group managing segments...
Dec 30 12:51:37.208596330 (597h22m37.079679376s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.240881687 (597h22m37.111964722s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.294293140 (597h22m37.165376175s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.302040142 (597h22m37.173123177s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.308919160 (597h22m37.180002196s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.370286944 (597h22m37.24136999s) [noise=true , address=77.90.31.10:4545 , piece_id=YQV3IPXMIBSO5ZCISFUBGRGHE6FIXLKTLNYKPWR42VNBKNOGPG5Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.397564325 (597h22m37.26864737s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.481372759 (597h22m37.352455794s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.868609147 (597h22m37.739692192s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.122711215 (597h22m37.993794251s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.156332275 (597h22m38.02741531s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.210830938 (597h22m38.081913983s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.318916249 (597h22m38.189999284s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.528107081 (597h22m38.399190127s) [noise=true , address=193.149.164.143:4694 , piece_id=AOZVLCCSBEYZX3WES7JJVOA2BECEBPII3A33PG5KCSQY6LSZ6Y3A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.528133793 (597h22m38.399216838s) [seg_pos={0 0}, upload=523642]: Segment reached optimal threshold of 14 pieces.
Dec 30 12:51:38.528364652 (597h22m38.399447687s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528373429 (597h22m38.399456464s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.528672969 (597h22m38.399756014s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528682567 (597h22m38.399765602s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.528839085 (597h22m38.39992212s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528847421 (597h22m38.399930476s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.529001655 (597h22m38.4000847s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection
Dec 30 12:51:38.529010362 (597h22m38.400093407s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.529029568 (597h22m38.400112603s) [seg_pos={0 0}, upload=523642]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 12:51:38.529107577 (597h22m38.400190612s) [upload=523642]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 12:51:38.529114079 (597h22m38.400197124s) [upload=523642]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 12:51:38.828356312 (597h22m38.699439387s) [upload=523642]: Done uploading segments...
Dec 30 12:51:38.828379166 (597h22m38.699462231s) [upload=523642]: Upload finished. err: <nil>
Duration: 36.31574601s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [.X..XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XX] duration:
5.001052606s trace:
https://jaeger.dev.storj.io/trace/4acfea1260a6bdf8
FAIL (2025-12-30 12:58:55 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001076331s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [......................X.......................X....X........] duration:
47.788490383s trace:
https://jaeger.dev.storj.io/trace/7b47a373389cd7c3
FAIL (2025-12-30 12:58:45 UTC)
orbiter warning: context deadline exceeded
Duration: 47.788508658s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [...XX..X.......X.XX..X..X.XX...............X..X.X..X....X...] duration:
6.140841237s trace:
https://jaeger.dev.storj.io/trace/23ad8207da0148f9
FAIL (2025-12-30 12:58:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ) context deadline exceeded
======LOGS======
ments...
Dec 30 12:55:52.163563927 (597h26m52.034646982s) [upload=523705]: Got next segment. Inline: false
Dec 30 12:55:52.163612819 (597h26m52.034695864s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 12:55:52.163618821 (597h26m52.034701866s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 12:55:52.410969244 (597h26m52.282052279s) [seg_pos={0 0}, upload=523705]: Begin upload segment...
Dec 30 12:55:52.411037163 (597h26m52.282120198s) [seg_pos={0 0}, upload=523705]: Done begin upload segment.
Dec 30 12:55:52.411066920 (597h26m52.282149955s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.411202477 (597h26m52.282285522s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.411721015 (597h26m52.28280405s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.412140103 (597h26m52.283223138s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.412521419 (597h26m52.283604454s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.412888228 (597h26m52.283971263s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.413237172 (597h26m52.284320207s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.413653144 (597h26m52.284736179s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.414092440 (597h26m52.285175475s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.415787007 (597h26m52.286870042s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.415936882 (597h26m52.287019917s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.416357675 (597h26m52.28744072s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.416794676 (597h26m52.287877721s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.417238962 (597h26m52.288321997s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.417715961 (597h26m52.288798996s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.418153584 (597h26m52.289236619s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:54.018730327 (597h26m53.889813362s) [upload=523705]: Next returned nil segment
Dec 30 12:55:54.018743952 (597h26m53.889826997s) [upload=523705]: Waiting for error group managing segments...
Dec 30 12:55:54.399128062 (597h26m54.270211097s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.399875084 (597h26m54.270958129s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.404232521 (597h26m54.275315556s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.424052321 (597h26m54.295135356s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.433077286 (597h26m54.304160321s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.437149530 (597h26m54.308232575s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.480458237 (597h26m54.351541282s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.689140229 (597h26m54.560223274s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.711163926 (597h26m54.582246961s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.730262152 (597h26m54.601345197s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.877886751 (597h26m54.748969796s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.889062301 (597h26m54.760145336s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:55.074468032 (597h26m54.945551077s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:55.074489473 (597h26m54.945572518s) [seg_pos={0 0}, upload=523705]: Segment reached optimal threshold of 13 pieces.
Dec 30 12:55:55.074727766 (597h26m54.945810811s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection
Dec 30 12:55:55.074735361 (597h26m54.945818396s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold
Dec 30 12:55:55.074903922 (597h26m54.945986967s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection
Dec 30 12:55:55.074929440 (597h26m54.946012485s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold
Dec 30 12:55:55.074969206 (597h26m54.946052241s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 30 12:55:55.074975178 (597h26m54.946058213s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold
Dec 30 12:55:55.075042837 (597h26m54.946125872s) [seg_pos={0 0}, upload=523705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 12:55:55.075106016 (597h26m54.946189051s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 12:55:55.075110695 (597h26m54.94619373s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 12:55:55.365579192 (597h26m55.236662227s) [upload=523705]: Done uploading segments...
Dec 30 12:55:55.365601594 (597h26m55.236684639s) [upload=523705]: Upload finished. err: <nil>
Duration: 6.14086819s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [..X..XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.X] duration:
5.001331639s trace:
https://jaeger.dev.storj.io/trace/43204983bb1bb812
FAIL (2025-12-30 12:58:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001355874s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...................................X........X...........X..X] duration:
30.001428823s trace:
https://jaeger.dev.storj.io/trace/17a3dfac1fc2c808
FAIL (2025-12-30 12:58:29 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001443701s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
7 out of 60 is failed (threshold=3) [................X.....X.X...X.........X....X........X.......] duration:
50.049807668s trace:
https://jaeger.dev.storj.io/trace/1e6433ec4a141391
FAIL (2025-12-30 12:58:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
[noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.352305080 (597h22m35.223388115s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.353083031 (597h22m35.224166066s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.353482140 (597h22m35.224565175s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.354315267 (597h22m35.225398302s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.354723675 (597h22m35.22580671s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.355352552 (597h22m35.226435587s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.356361994 (597h22m35.227445029s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.356578216 (597h22m35.227661261s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.357840370 (597h22m35.228923405s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.359039433 (597h22m35.230122468s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.359333493 (597h22m35.230416538s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.360694154 (597h22m35.231777189s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.361764952 (597h22m35.232847987s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.363365710 (597h22m35.234448755s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.363684648 (597h22m35.234767683s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:36.749009517 (597h22m36.620092552s) [upload=523642]: Next returned nil segment
Dec 30 12:51:36.749023393 (597h22m36.620106428s) [upload=523642]: Waiting for error group managing segments...
Dec 30 12:51:37.208596330 (597h22m37.079679376s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.240881687 (597h22m37.111964722s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.294293140 (597h22m37.165376175s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.302040142 (597h22m37.173123177s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.308919160 (597h22m37.180002196s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.370286944 (597h22m37.24136999s) [noise=true , address=77.90.31.10:4545 , piece_id=YQV3IPXMIBSO5ZCISFUBGRGHE6FIXLKTLNYKPWR42VNBKNOGPG5Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.397564325 (597h22m37.26864737s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.481372759 (597h22m37.352455794s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.868609147 (597h22m37.739692192s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.122711215 (597h22m37.993794251s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.156332275 (597h22m38.02741531s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.210830938 (597h22m38.081913983s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.318916249 (597h22m38.189999284s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.528107081 (597h22m38.399190127s) [noise=true , address=193.149.164.143:4694 , piece_id=AOZVLCCSBEYZX3WES7JJVOA2BECEBPII3A33PG5KCSQY6LSZ6Y3A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.528133793 (597h22m38.399216838s) [seg_pos={0 0}, upload=523642]: Segment reached optimal threshold of 14 pieces.
Dec 30 12:51:38.528364652 (597h22m38.399447687s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528373429 (597h22m38.399456464s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.528672969 (597h22m38.399756014s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528682567 (597h22m38.399765602s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.528839085 (597h22m38.39992212s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528847421 (597h22m38.399930476s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.529001655 (597h22m38.4000847s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection
Dec 30 12:51:38.529010362 (597h22m38.400093407s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.529029568 (597h22m38.400112603s) [seg_pos={0 0}, upload=523642]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 12:51:38.529107577 (597h22m38.400190612s) [upload=523642]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 12:51:38.529114079 (597h22m38.400197124s) [upload=523642]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 12:51:38.828356312 (597h22m38.699439387s) [upload=523642]: Done uploading segments...
Dec 30 12:51:38.828379166 (597h22m38.699462231s) [upload=523642]: Upload finished. err: <nil>
Duration: 50.049834068s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....................................X........X...........X..] duration:
17.306343538s trace:
https://jaeger.dev.storj.io/trace/4fe53921b90940f1
FAIL (2025-12-30 12:57:41 UTC)
orbiter warning: context deadline exceeded
Duration: 17.306375088s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.......................X.......................X....X.......] duration:
16.594417122s trace:
https://jaeger.dev.storj.io/trace/7b47a373389cd7c3
FAIL (2025-12-30 12:57:33 UTC)
orbiter warning: context deadline exceeded
Duration: 16.594441388s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [....XX..X.......X.XX..X..X.XX...............X..X.X..X....X..] duration:
5.783396756s trace:
https://jaeger.dev.storj.io/trace/23ad8207da0148f9
FAIL (2025-12-30 12:57:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ) context deadline exceeded
======LOGS======
ments...
Dec 30 12:55:52.163563927 (597h26m52.034646982s) [upload=523705]: Got next segment. Inline: false
Dec 30 12:55:52.163612819 (597h26m52.034695864s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 12:55:52.163618821 (597h26m52.034701866s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 12:55:52.410969244 (597h26m52.282052279s) [seg_pos={0 0}, upload=523705]: Begin upload segment...
Dec 30 12:55:52.411037163 (597h26m52.282120198s) [seg_pos={0 0}, upload=523705]: Done begin upload segment.
Dec 30 12:55:52.411066920 (597h26m52.282149955s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.411202477 (597h26m52.282285522s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.411721015 (597h26m52.28280405s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.412140103 (597h26m52.283223138s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.412521419 (597h26m52.283604454s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.412888228 (597h26m52.283971263s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.413237172 (597h26m52.284320207s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.413653144 (597h26m52.284736179s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.414092440 (597h26m52.285175475s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.415787007 (597h26m52.286870042s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.415936882 (597h26m52.287019917s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.416357675 (597h26m52.28744072s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.416794676 (597h26m52.287877721s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.417238962 (597h26m52.288321997s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.417715961 (597h26m52.288798996s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.418153584 (597h26m52.289236619s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:54.018730327 (597h26m53.889813362s) [upload=523705]: Next returned nil segment
Dec 30 12:55:54.018743952 (597h26m53.889826997s) [upload=523705]: Waiting for error group managing segments...
Dec 30 12:55:54.399128062 (597h26m54.270211097s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.399875084 (597h26m54.270958129s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.404232521 (597h26m54.275315556s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.424052321 (597h26m54.295135356s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.433077286 (597h26m54.304160321s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.437149530 (597h26m54.308232575s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.480458237 (597h26m54.351541282s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.689140229 (597h26m54.560223274s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.711163926 (597h26m54.582246961s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.730262152 (597h26m54.601345197s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.877886751 (597h26m54.748969796s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.889062301 (597h26m54.760145336s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:55.074468032 (597h26m54.945551077s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:55.074489473 (597h26m54.945572518s) [seg_pos={0 0}, upload=523705]: Segment reached optimal threshold of 13 pieces.
Dec 30 12:55:55.074727766 (597h26m54.945810811s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection
Dec 30 12:55:55.074735361 (597h26m54.945818396s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold
Dec 30 12:55:55.074903922 (597h26m54.945986967s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection
Dec 30 12:55:55.074929440 (597h26m54.946012485s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold
Dec 30 12:55:55.074969206 (597h26m54.946052241s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 30 12:55:55.074975178 (597h26m54.946058213s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold
Dec 30 12:55:55.075042837 (597h26m54.946125872s) [seg_pos={0 0}, upload=523705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 12:55:55.075106016 (597h26m54.946189051s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 12:55:55.075110695 (597h26m54.94619373s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 12:55:55.365579192 (597h26m55.236662227s) [upload=523705]: Done uploading segments...
Dec 30 12:55:55.365601594 (597h26m55.236684639s) [upload=523705]: Upload finished. err: <nil>
Duration: 5.783426603s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [X..X..XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.] duration:
3.666302156s trace:
https://jaeger.dev.storj.io/trace/7b47da8923bfb651
FAIL (2025-12-30 12:57:28 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.666329097s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....................................X........X...........X.] duration:
26.65693263s trace:
https://jaeger.dev.storj.io/trace/4fe53921b90940f1
FAIL (2025-12-30 12:57:17 UTC)
orbiter warning: context deadline exceeded
Duration: 26.656959272s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [.X..X..XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX] duration:
5.001612655s trace:
https://jaeger.dev.storj.io/trace/7b47da8923bfb651
FAIL (2025-12-30 12:57:10 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001638734s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [........................X.......................X....X......] duration:
16.970545533s trace:
https://jaeger.dev.storj.io/trace/7b47a373389cd7c3
FAIL (2025-12-30 12:57:01 UTC)
orbiter warning: context deadline exceeded
Duration: 16.970574848s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
7 out of 60 is failed (threshold=3) [.................X.....X.X...X.........X....X........X......] duration:
29.525820993s trace:
https://jaeger.dev.storj.io/trace/1e6433ec4a141391
FAIL (2025-12-30 12:56:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
[noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.352305080 (597h22m35.223388115s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.353083031 (597h22m35.224166066s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.353482140 (597h22m35.224565175s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.354315267 (597h22m35.225398302s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.354723675 (597h22m35.22580671s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.355352552 (597h22m35.226435587s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.356361994 (597h22m35.227445029s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.356578216 (597h22m35.227661261s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.357840370 (597h22m35.228923405s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.359039433 (597h22m35.230122468s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.359333493 (597h22m35.230416538s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.360694154 (597h22m35.231777189s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.361764952 (597h22m35.232847987s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.363365710 (597h22m35.234448755s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.363684648 (597h22m35.234767683s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:36.749009517 (597h22m36.620092552s) [upload=523642]: Next returned nil segment
Dec 30 12:51:36.749023393 (597h22m36.620106428s) [upload=523642]: Waiting for error group managing segments...
Dec 30 12:51:37.208596330 (597h22m37.079679376s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.240881687 (597h22m37.111964722s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.294293140 (597h22m37.165376175s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.302040142 (597h22m37.173123177s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.308919160 (597h22m37.180002196s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.370286944 (597h22m37.24136999s) [noise=true , address=77.90.31.10:4545 , piece_id=YQV3IPXMIBSO5ZCISFUBGRGHE6FIXLKTLNYKPWR42VNBKNOGPG5Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.397564325 (597h22m37.26864737s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.481372759 (597h22m37.352455794s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.868609147 (597h22m37.739692192s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.122711215 (597h22m37.993794251s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.156332275 (597h22m38.02741531s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.210830938 (597h22m38.081913983s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.318916249 (597h22m38.189999284s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.528107081 (597h22m38.399190127s) [noise=true , address=193.149.164.143:4694 , piece_id=AOZVLCCSBEYZX3WES7JJVOA2BECEBPII3A33PG5KCSQY6LSZ6Y3A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.528133793 (597h22m38.399216838s) [seg_pos={0 0}, upload=523642]: Segment reached optimal threshold of 14 pieces.
Dec 30 12:51:38.528364652 (597h22m38.399447687s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528373429 (597h22m38.399456464s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.528672969 (597h22m38.399756014s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528682567 (597h22m38.399765602s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.528839085 (597h22m38.39992212s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528847421 (597h22m38.399930476s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.529001655 (597h22m38.4000847s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection
Dec 30 12:51:38.529010362 (597h22m38.400093407s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.529029568 (597h22m38.400112603s) [seg_pos={0 0}, upload=523642]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 12:51:38.529107577 (597h22m38.400190612s) [upload=523642]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 12:51:38.529114079 (597h22m38.400197124s) [upload=523642]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 12:51:38.828356312 (597h22m38.699439387s) [upload=523642]: Done uploading segments...
Dec 30 12:51:38.828379166 (597h22m38.699462231s) [upload=523642]: Upload finished. err: <nil>
Duration: 29.525838978s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [X....XX..X.......X.XX..X..X.XX...............X..X.X..X....X.] duration:
5.781072883s trace:
https://jaeger.dev.storj.io/trace/23ad8207da0148f9
FAIL (2025-12-30 12:56:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ) context deadline exceeded
======LOGS======
ments...
Dec 30 12:55:52.163563927 (597h26m52.034646982s) [upload=523705]: Got next segment. Inline: false
Dec 30 12:55:52.163612819 (597h26m52.034695864s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 12:55:52.163618821 (597h26m52.034701866s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 12:55:52.410969244 (597h26m52.282052279s) [seg_pos={0 0}, upload=523705]: Begin upload segment...
Dec 30 12:55:52.411037163 (597h26m52.282120198s) [seg_pos={0 0}, upload=523705]: Done begin upload segment.
Dec 30 12:55:52.411066920 (597h26m52.282149955s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.411202477 (597h26m52.282285522s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.411721015 (597h26m52.28280405s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.412140103 (597h26m52.283223138s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.412521419 (597h26m52.283604454s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.412888228 (597h26m52.283971263s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.413237172 (597h26m52.284320207s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.413653144 (597h26m52.284736179s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.414092440 (597h26m52.285175475s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.415787007 (597h26m52.286870042s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.415936882 (597h26m52.287019917s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.416357675 (597h26m52.28744072s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.416794676 (597h26m52.287877721s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.417238962 (597h26m52.288321997s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.417715961 (597h26m52.288798996s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.418153584 (597h26m52.289236619s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:54.018730327 (597h26m53.889813362s) [upload=523705]: Next returned nil segment
Dec 30 12:55:54.018743952 (597h26m53.889826997s) [upload=523705]: Waiting for error group managing segments...
Dec 30 12:55:54.399128062 (597h26m54.270211097s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.399875084 (597h26m54.270958129s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.404232521 (597h26m54.275315556s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.424052321 (597h26m54.295135356s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.433077286 (597h26m54.304160321s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.437149530 (597h26m54.308232575s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.480458237 (597h26m54.351541282s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.689140229 (597h26m54.560223274s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.711163926 (597h26m54.582246961s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.730262152 (597h26m54.601345197s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.877886751 (597h26m54.748969796s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.889062301 (597h26m54.760145336s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:55.074468032 (597h26m54.945551077s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:55.074489473 (597h26m54.945572518s) [seg_pos={0 0}, upload=523705]: Segment reached optimal threshold of 13 pieces.
Dec 30 12:55:55.074727766 (597h26m54.945810811s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection
Dec 30 12:55:55.074735361 (597h26m54.945818396s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold
Dec 30 12:55:55.074903922 (597h26m54.945986967s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection
Dec 30 12:55:55.074929440 (597h26m54.946012485s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold
Dec 30 12:55:55.074969206 (597h26m54.946052241s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 30 12:55:55.074975178 (597h26m54.946058213s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold
Dec 30 12:55:55.075042837 (597h26m54.946125872s) [seg_pos={0 0}, upload=523705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 12:55:55.075106016 (597h26m54.946189051s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 12:55:55.075110695 (597h26m54.94619373s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 12:55:55.365579192 (597h26m55.236662227s) [upload=523705]: Done uploading segments...
Dec 30 12:55:55.365601594 (597h26m55.236684639s) [upload=523705]: Upload finished. err: <nil>
Duration: 5.781098844s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.........................X.......................X....X.....] duration:
21.895188653s trace:
https://jaeger.dev.storj.io/trace/7b47a373389cd7c3
FAIL (2025-12-30 12:56:43 UTC)
orbiter warning: context deadline exceeded
Duration: 21.89521305s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [..X..X..XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XX] duration:
5.000748889s trace:
https://jaeger.dev.storj.io/trace/534a65ad0476b03f
FAIL (2025-12-30 12:56:42 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000768657s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......................................X........X...........X] duration:
30.001047284s trace:
https://jaeger.dev.storj.io/trace/4fe53921b90940f1
FAIL (2025-12-30 12:56:31 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001067171s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
7 out of 60 is failed (threshold=3) [..................X.....X.X...X.........X....X........X.....] duration:
35.465864449s trace:
https://jaeger.dev.storj.io/trace/1e6433ec4a141391
FAIL (2025-12-30 12:56:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
[noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.352305080 (597h22m35.223388115s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.353083031 (597h22m35.224166066s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.353482140 (597h22m35.224565175s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.354315267 (597h22m35.225398302s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.354723675 (597h22m35.22580671s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.355352552 (597h22m35.226435587s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.356361994 (597h22m35.227445029s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.356578216 (597h22m35.227661261s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.357840370 (597h22m35.228923405s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.359039433 (597h22m35.230122468s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.359333493 (597h22m35.230416538s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.360694154 (597h22m35.231777189s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.361764952 (597h22m35.232847987s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.363365710 (597h22m35.234448755s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.363684648 (597h22m35.234767683s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:36.749009517 (597h22m36.620092552s) [upload=523642]: Next returned nil segment
Dec 30 12:51:36.749023393 (597h22m36.620106428s) [upload=523642]: Waiting for error group managing segments...
Dec 30 12:51:37.208596330 (597h22m37.079679376s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.240881687 (597h22m37.111964722s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.294293140 (597h22m37.165376175s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.302040142 (597h22m37.173123177s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.308919160 (597h22m37.180002196s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.370286944 (597h22m37.24136999s) [noise=true , address=77.90.31.10:4545 , piece_id=YQV3IPXMIBSO5ZCISFUBGRGHE6FIXLKTLNYKPWR42VNBKNOGPG5Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.397564325 (597h22m37.26864737s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.481372759 (597h22m37.352455794s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.868609147 (597h22m37.739692192s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.122711215 (597h22m37.993794251s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.156332275 (597h22m38.02741531s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.210830938 (597h22m38.081913983s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.318916249 (597h22m38.189999284s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.528107081 (597h22m38.399190127s) [noise=true , address=193.149.164.143:4694 , piece_id=AOZVLCCSBEYZX3WES7JJVOA2BECEBPII3A33PG5KCSQY6LSZ6Y3A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.528133793 (597h22m38.399216838s) [seg_pos={0 0}, upload=523642]: Segment reached optimal threshold of 14 pieces.
Dec 30 12:51:38.528364652 (597h22m38.399447687s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528373429 (597h22m38.399456464s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.528672969 (597h22m38.399756014s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528682567 (597h22m38.399765602s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.528839085 (597h22m38.39992212s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528847421 (597h22m38.399930476s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.529001655 (597h22m38.4000847s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection
Dec 30 12:51:38.529010362 (597h22m38.400093407s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.529029568 (597h22m38.400112603s) [seg_pos={0 0}, upload=523642]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 12:51:38.529107577 (597h22m38.400190612s) [upload=523642]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 12:51:38.529114079 (597h22m38.400197124s) [upload=523642]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 12:51:38.828356312 (597h22m38.699439387s) [upload=523642]: Done uploading segments...
Dec 30 12:51:38.828379166 (597h22m38.699462231s) [upload=523642]: Upload finished. err: <nil>
Duration: 35.465880819s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [XX....XX..X.......X.XX..X..X.XX...............X..X.X..X....X] duration:
30.002286469s trace:
https://jaeger.dev.storj.io/trace/23ad8207da0148f9
FAIL (2025-12-30 12:56:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ) context deadline exceeded
======LOGS======
ments...
Dec 30 12:55:52.163563927 (597h26m52.034646982s) [upload=523705]: Got next segment. Inline: false
Dec 30 12:55:52.163612819 (597h26m52.034695864s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 12:55:52.163618821 (597h26m52.034701866s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 12:55:52.410969244 (597h26m52.282052279s) [seg_pos={0 0}, upload=523705]: Begin upload segment...
Dec 30 12:55:52.411037163 (597h26m52.282120198s) [seg_pos={0 0}, upload=523705]: Done begin upload segment.
Dec 30 12:55:52.411066920 (597h26m52.282149955s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.411202477 (597h26m52.282285522s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.411721015 (597h26m52.28280405s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.412140103 (597h26m52.283223138s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.412521419 (597h26m52.283604454s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.412888228 (597h26m52.283971263s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.413237172 (597h26m52.284320207s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.413653144 (597h26m52.284736179s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.414092440 (597h26m52.285175475s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.415787007 (597h26m52.286870042s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.415936882 (597h26m52.287019917s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.416357675 (597h26m52.28744072s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.416794676 (597h26m52.287877721s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.417238962 (597h26m52.288321997s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.417715961 (597h26m52.288798996s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:52.418153584 (597h26m52.289236619s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Uploading piece...
Dec 30 12:55:54.018730327 (597h26m53.889813362s) [upload=523705]: Next returned nil segment
Dec 30 12:55:54.018743952 (597h26m53.889826997s) [upload=523705]: Waiting for error group managing segments...
Dec 30 12:55:54.399128062 (597h26m54.270211097s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.399875084 (597h26m54.270958129s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.404232521 (597h26m54.275315556s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.424052321 (597h26m54.295135356s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.433077286 (597h26m54.304160321s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.437149530 (597h26m54.308232575s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.480458237 (597h26m54.351541282s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.689140229 (597h26m54.560223274s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.711163926 (597h26m54.582246961s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.730262152 (597h26m54.601345197s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.877886751 (597h26m54.748969796s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:54.889062301 (597h26m54.760145336s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:55.074468032 (597h26m54.945551077s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil>
Dec 30 12:55:55.074489473 (597h26m54.945572518s) [seg_pos={0 0}, upload=523705]: Segment reached optimal threshold of 13 pieces.
Dec 30 12:55:55.074727766 (597h26m54.945810811s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection
Dec 30 12:55:55.074735361 (597h26m54.945818396s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold
Dec 30 12:55:55.074903922 (597h26m54.945986967s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection
Dec 30 12:55:55.074929440 (597h26m54.946012485s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold
Dec 30 12:55:55.074969206 (597h26m54.946052241s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 30 12:55:55.074975178 (597h26m54.946058213s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold
Dec 30 12:55:55.075042837 (597h26m54.946125872s) [seg_pos={0 0}, upload=523705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 12:55:55.075106016 (597h26m54.946189051s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 12:55:55.075110695 (597h26m54.94619373s) [upload=523705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 12:55:55.365579192 (597h26m55.236662227s) [upload=523705]: Done uploading segments...
Dec 30 12:55:55.365601594 (597h26m55.236684639s) [upload=523705]: Upload finished. err: <nil>
Duration: 30.002396509s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..........................X.......................X....X....] duration:
6.510648399s trace:
https://jaeger.dev.storj.io/trace/7b47a373389cd7c3
FAIL (2025-12-30 12:56:10 UTC)
orbiter warning: context deadline exceeded
Duration: 6.510677715s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [X..X..X..XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.X] duration:
5.001657039s trace:
https://jaeger.dev.storj.io/trace/998b3c47e703ed7
FAIL (2025-12-30 12:55:57 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001683348s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
8 out of 60 is failed (threshold=3) [X..................X.....X.X...X.........X....X........X....] duration:
48.523168031s trace:
https://jaeger.dev.storj.io/trace/1e6433ec4a141391
FAIL (2025-12-30 12:55:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
[noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.352305080 (597h22m35.223388115s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.353083031 (597h22m35.224166066s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.353482140 (597h22m35.224565175s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.354315267 (597h22m35.225398302s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.354723675 (597h22m35.22580671s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.355352552 (597h22m35.226435587s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.356361994 (597h22m35.227445029s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.356578216 (597h22m35.227661261s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.357840370 (597h22m35.228923405s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.359039433 (597h22m35.230122468s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.359333493 (597h22m35.230416538s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.360694154 (597h22m35.231777189s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.361764952 (597h22m35.232847987s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.363365710 (597h22m35.234448755s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:35.363684648 (597h22m35.234767683s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Uploading piece...
Dec 30 12:51:36.749009517 (597h22m36.620092552s) [upload=523642]: Next returned nil segment
Dec 30 12:51:36.749023393 (597h22m36.620106428s) [upload=523642]: Waiting for error group managing segments...
Dec 30 12:51:37.208596330 (597h22m37.079679376s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.240881687 (597h22m37.111964722s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.294293140 (597h22m37.165376175s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.302040142 (597h22m37.173123177s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.308919160 (597h22m37.180002196s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.370286944 (597h22m37.24136999s) [noise=true , address=77.90.31.10:4545 , piece_id=YQV3IPXMIBSO5ZCISFUBGRGHE6FIXLKTLNYKPWR42VNBKNOGPG5Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.397564325 (597h22m37.26864737s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.481372759 (597h22m37.352455794s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:37.868609147 (597h22m37.739692192s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.122711215 (597h22m37.993794251s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.156332275 (597h22m38.02741531s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.210830938 (597h22m38.081913983s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.318916249 (597h22m38.189999284s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.528107081 (597h22m38.399190127s) [noise=true , address=193.149.164.143:4694 , piece_id=AOZVLCCSBEYZX3WES7JJVOA2BECEBPII3A33PG5KCSQY6LSZ6Y3A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil>
Dec 30 12:51:38.528133793 (597h22m38.399216838s) [seg_pos={0 0}, upload=523642]: Segment reached optimal threshold of 14 pieces.
Dec 30 12:51:38.528364652 (597h22m38.399447687s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528373429 (597h22m38.399456464s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.528672969 (597h22m38.399756014s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528682567 (597h22m38.399765602s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.528839085 (597h22m38.39992212s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF
Dec 30 12:51:38.528847421 (597h22m38.399930476s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.529001655 (597h22m38.4000847s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection
Dec 30 12:51:38.529010362 (597h22m38.400093407s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold
Dec 30 12:51:38.529029568 (597h22m38.400112603s) [seg_pos={0 0}, upload=523642]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 12:51:38.529107577 (597h22m38.400190612s) [upload=523642]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 12:51:38.529114079 (597h22m38.400197124s) [upload=523642]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 12:51:38.828356312 (597h22m38.699439387s) [upload=523642]: Done uploading segments...
Dec 30 12:51:38.828379166 (597h22m38.699462231s) [upload=523642]: Upload finished. err: <nil>
Duration: 48.52319929s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................X........X...........] duration:
8.445352275s trace:
https://jaeger.dev.storj.io/trace/5287a23df1c13acc
FAIL (2025-12-30 12:55:36 UTC)
orbiter warning: context deadline exceeded
Duration: 8.445376551s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [.X..X..X..XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.] duration:
3.67396163s trace:
https://jaeger.dev.storj.io/trace/67c673269acb4aa
FAIL (2025-12-30 12:55:31 UTC)
orbiter warning: context deadline exceeded
Duration: 3.673991827s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................X........X..........] duration:
14.951779272s trace:
https://jaeger.dev.storj.io/trace/5287a23df1c13acc
FAIL (2025-12-30 12:55:23 UTC)
orbiter warning: context deadline exceeded
Duration: 14.951806584s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...........................X.......................X....X...] duration:
15.90983094s trace:
https://jaeger.dev.storj.io/trace/7b47a373389cd7c3
FAIL (2025-12-30 12:55:16 UTC)
orbiter warning: context deadline exceeded
Duration: 15.90985753s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.XX....XX..X.......X.XX..X..X.XX...............X..X.X..X....] duration:
5.836961433s trace:
https://jaeger.dev.storj.io/trace/7e4805d173e0a369
FAIL (2025-12-30 12:55:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: KUNKLOKIRSYDCFDOMUXCUUBGR4CZTZUKSRLKLZI5NLBD5QMNGTCQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: IZTI4X4FSVD3K563B5VKKZDAHMUBW2ZDQHZATZNOZMSKKZM7FXGA) context deadline exceeded
======LOGS======
Dec 30 12:52:14.115932176 (597h23m13.987015211s) [upload=523651]: Starting upload
Dec 30 12:52:14.115994284 (597h23m13.987077329s) [upload=523651]: Done starting upload
Dec 30 12:52:14.116134391 (597h23m13.987217436s) [upload=523651]: Uploading segments...
Dec 30 12:52:14.116323982 (597h23m13.987407027s) [upload=523651]: Got next segment. Inline: false
Dec 30 12:52:14.116390138 (597h23m13.987473173s) [upload=523651]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 12:52:14.116396260 (597h23m13.987479315s) [upload=523651]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 12:52:14.358767612 (597h23m14.229850647s) [seg_pos={0 0}, upload=523651]: Begin upload segment...
Dec 30 12:52:14.358813980 (597h23m14.229897015s) [seg_pos={0 0}, upload=523651]: Done begin upload segment.
Dec 30 12:52:14.358844779 (597h23m14.229927814s) [noise=true , address=145.79.227.10:4401 , piece_id=M56P2SXBJSURDWMWYZJLHZWQZEIEMNUURQ466SZANLUTHX4REE5Q, seg_pos={0 0}, upload=523651]: Uploading piece...
Dec 30 12:52:14.358995876 (597h23m14.230078911s) [noise=true , address=145.79.227.41:4087 , piece_id=YRQHXM2SXDBB55WWMHJRU4WNMCSNWM3ECQKLTCUK6LDIQUZHJYRQ, seg_pos={0 0}, upload=523651]: Uploading piece...
Dec 30 12:52:14.359937319 (597h23m14.231020364s) [noise=true , address=77.90.31.10:4518 , piece_id=KUNKLOKIRSYDCFDOMUXCUUBGR4CZTZUKSRLKLZI5NLBD5QMNGTCQ, seg_pos={0 0}, upload=523651]: Uploading piece...
Dec 30 12:52:14.360144694 (597h23m14.231227739s) [noise=true , address=145.79.227.21:4473 , piece_id=H2BVWUFZGS5UY3CYE5EWKYMXX6LX3UZ735N3ER3EQNA7WTUYLLEA, seg_pos={0 0}, upload=523651]: Uploading piece...
Dec 30 12:52:14.360694351 (597h23m14.231777396s) [noise=true , address=77.90.31.14:4661 , piece_id=CSWEMPTKJVW4GZC5DLDZUY5J43UALWK2UT7CY7MJTDKYUAJXCL7A, seg_pos={0 0}, upload=523651]: Uploading piece...
Dec 30 12:52:14.361171960 (597h23m14.232254995s) [noise=true , address=145.79.227.22:4510 , piece_id=QWT3E76ALSUOQITBKE4O2E5DNZE52IVGUECTIVOV44I7MNXXY6NA, seg_pos={0 0}, upload=523651]: Uploading piece...
Dec 30 12:52:14.361831737 (597h23m14.232914782s) [noise=true , address=145.79.227.13:4804 , piece_id=4YDWGK35ZFVWPKGQDT4A5EBT23S5CBU6QANWEXZFDQK54C4S4QUQ, seg_pos={0 0}, upload=523651]: Uploading piece...
Dec 30 12:52:14.362325757 (597h23m14.233408802s) [noise=true , address=145.79.227.30:4761 , piece_id=EVMZUUTYJKJRDHAONOCRJRZQL54FIMWWKTFPS5KVYUZAOCHL4XOQ, seg_pos={0 0}, upload=523651]: Uploading piece...
Dec 30 12:52:14.362909749 (597h23m14.233992794s) [noise=true , address=145.79.227.28:4726 , piece_id=4B2OFKLPN3YIH46YSYNXXE6NVUM2AB7KEYQOAZYQMS2Z4SKTDUYA, seg_pos={0 0}, upload=523651]: Uploading piece...
Dec 30 12:52:14.363590836 (597h23m14.234673881s) [noise=true , address=145.79.227.14:4838 , piece_id=WH2PMMJVZMBXKHUX3X3RGVD4IDVCZ4EYY3JBVOV7F2YMAHPFI2DA, seg_pos={0 0}, upload=523651]: Uploading piece...
Dec 30 12:52:14.366121074 (597h23m14.237204149s) [noise=true , address=45.91.116.9:4263 , piece_id=HEOOYEEFNXNRGOQ45S6DPNNT5YZIZWIUG2VH4VKWLD7K62NP4ZQQ, seg_pos={0 0}, upload=523651]: Uploading piece...
Dec 30 12:52:14.366652215 (597h23m14.23773526s) [noise=true , address=145.79.227.15:4874 , piece_id=3R5H6XZYTULAWDWHTAN65I7PPPHOFXD5RNM35ZVUVYNH545YTXGQ, seg_pos={0 0}, upload=523651]: Uploading piece...
Dec 30 12:52:14.367079769 (597h23m14.238162804s) [noise=true , address=145.79.227.19:4078 , piece_id=MQIQVZ4KSF3ZQZEBS3ISSC2C4ETEBE2ZLQYQA3G2DOFD5C6FEUJA, seg_pos={0 0}, upload=523651]: Uploading piece...
Dec 30 12:52:14.367563110 (597h23m14.238646145s) [noise=true , address=45.91.116.10:4083 , piece_id=IZTI4X4FSVD3K563B5VKKZDAHMUBW2ZDQHZATZNOZMSKKZM7FXGA, seg_pos={0 0}, upload=523651]: Uploading piece...
Dec 30 12:52:14.368083380 (597h23m14.239166415s) [noise=true , address=145.79.227.34:4905 , piece_id=JEEM7QYBUR7X7675WIJJG4PTLSZR53BL45HV6MZKTDD65KCFDLRQ, seg_pos={0 0}, upload=523651]: Uploading piece...
Dec 30 12:52:14.370335168 (597h23m14.241418213s) [noise=true , address=145.79.227.29:4943 , piece_id=JPE7GTBBRY54NABCPQJFJJMN3YEVY3RLODE6JIFTEMMNKUL3RMIA, seg_pos={0 0}, upload=523651]: Uploading piece...
Dec 30 12:52:15.971132591 (597h23m15.842215636s) [upload=523651]: Next returned nil segment
Dec 30 12:52:15.971151498 (597h23m15.842234533s) [upload=523651]: Waiting for error group managing segments...
Dec 30 12:52:16.352891969 (597h23m16.223975014s) [noise=true , address=145.79.227.22:4510 , piece_id=QWT3E76ALSUOQITBKE4O2E5DNZE52IVGUECTIVOV44I7MNXXY6NA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil>
Dec 30 12:52:16.353001558 (597h23m16.224084593s) [noise=true , address=145.79.227.13:4804 , piece_id=4YDWGK35ZFVWPKGQDT4A5EBT23S5CBU6QANWEXZFDQK54C4S4QUQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil>
Dec 30 12:52:16.360107449 (597h23m16.231190504s) [noise=true , address=145.79.227.19:4078 , piece_id=MQIQVZ4KSF3ZQZEBS3ISSC2C4ETEBE2ZLQYQA3G2DOFD5C6FEUJA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil>
Dec 30 12:52:16.361588549 (597h23m16.232671594s) [noise=true , address=145.79.227.10:4401 , piece_id=M56P2SXBJSURDWMWYZJLHZWQZEIEMNUURQ466SZANLUTHX4REE5Q, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil>
Dec 30 12:52:16.370586382 (597h23m16.241669427s) [noise=true , address=145.79.227.28:4726 , piece_id=4B2OFKLPN3YIH46YSYNXXE6NVUM2AB7KEYQOAZYQMS2Z4SKTDUYA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil>
Dec 30 12:52:16.379901019 (597h23m16.250984054s) [noise=true , address=145.79.227.21:4473 , piece_id=H2BVWUFZGS5UY3CYE5EWKYMXX6LX3UZ735N3ER3EQNA7WTUYLLEA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil>
Dec 30 12:52:16.381883654 (597h23m16.252966699s) [noise=true , address=145.79.227.14:4838 , piece_id=WH2PMMJVZMBXKHUX3X3RGVD4IDVCZ4EYY3JBVOV7F2YMAHPFI2DA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil>
Dec 30 12:52:16.382184477 (597h23m16.253267512s) [noise=true , address=145.79.227.41:4087 , piece_id=YRQHXM2SXDBB55WWMHJRU4WNMCSNWM3ECQKLTCUK6LDIQUZHJYRQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil>
Dec 30 12:52:16.386081047 (597h23m16.257164082s) [noise=true , address=145.79.227.15:4874 , piece_id=3R5H6XZYTULAWDWHTAN65I7PPPHOFXD5RNM35ZVUVYNH545YTXGQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil>
Dec 30 12:52:16.648851922 (597h23m16.519934967s) [noise=true , address=77.90.31.14:4661 , piece_id=CSWEMPTKJVW4GZC5DLDZUY5J43UALWK2UT7CY7MJTDKYUAJXCL7A, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil>
Dec 30 12:52:16.690922852 (597h23m16.562005897s) [noise=true , address=77.90.31.10:4518 , piece_id=KUNKLOKIRSYDCFDOMUXCUUBGR4CZTZUKSRLKLZI5NLBD5QMNGTCQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil>
Dec 30 12:52:16.814557211 (597h23m16.685640246s) [noise=true , address=45.91.116.9:4263 , piece_id=HEOOYEEFNXNRGOQ45S6DPNNT5YZIZWIUG2VH4VKWLD7K62NP4ZQQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil>
Dec 30 12:52:17.475466253 (597h23m17.346549298s) [noise=true , address=45.91.116.10:4083 , piece_id=IZTI4X4FSVD3K563B5VKKZDAHMUBW2ZDQHZATZNOZMSKKZM7FXGA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil>
Dec 30 12:52:17.475491310 (597h23m17.346574355s) [seg_pos={0 0}, upload=523651]: Segment reached optimal threshold of 13 pieces.
Dec 30 12:52:17.475700448 (597h23m17.346783493s) [noise=true , address=145.79.227.34:4905 , piece_id=JEEM7QYBUR7X7675WIJJG4PTLSZR53BL45HV6MZKTDD65KCFDLRQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 30 12:52:17.475708314 (597h23m17.346791359s) [noise=true , address=145.79.227.34:4905 , piece_id=JEEM7QYBUR7X7675WIJJG4PTLSZR53BL45HV6MZKTDD65KCFDLRQ, seg_pos={0 0}, upload=523651]: Long tail cancellation due to optimal threshold
Dec 30 12:52:17.476027872 (597h23m17.347110907s) [noise=true , address=145.79.227.29:4943 , piece_id=JPE7GTBBRY54NABCPQJFJJMN3YEVY3RLODE6JIFTEMMNKUL3RMIA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF
Dec 30 12:52:17.476036999 (597h23m17.347120034s) [noise=true , address=145.79.227.29:4943 , piece_id=JPE7GTBBRY54NABCPQJFJJMN3YEVY3RLODE6JIFTEMMNKUL3RMIA, seg_pos={0 0}, upload=523651]: Long tail cancellation due to optimal threshold
Dec 30 12:52:17.476181614 (597h23m17.347264659s) [noise=true , address=145.79.227.30:4761 , piece_id=EVMZUUTYJKJRDHAONOCRJRZQL54FIMWWKTFPS5KVYUZAOCHL4XOQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 30 12:52:17.476189659 (597h23m17.347272705s) [noise=true , address=145.79.227.30:4761 , piece_id=EVMZUUTYJKJRDHAONOCRJRZQL54FIMWWKTFPS5KVYUZAOCHL4XOQ, seg_pos={0 0}, upload=523651]: Long tail cancellation due to optimal threshold
Dec 30 12:52:17.476211431 (597h23m17.347294476s) [seg_pos={0 0}, upload=523651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 12:52:17.476287837 (597h23m17.347370882s) [upload=523651]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 12:52:17.476293798 (597h23m17.347376843s) [upload=523651]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 12:52:17.763390204 (597h23m17.634473239s) [upload=523651]: Done uploading segments...
Dec 30 12:52:17.763411815 (597h23m17.63449485s) [upload=523651]: Upload finished. err: <nil>
Duration: 5.836993985s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [..X..X..X..XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X] duration:
5.001436772s trace:
https://jaeger.dev.storj.io/trace/67c673269acb4aa
FAIL (2025-12-30 12:54:59 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001457472s
core
uplink
us-select-1
us1-placement-20
|