Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
36 out of 60 is failed (threshold=6) [XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....X.X.XXX.] duration:
3.594123698s trace:
https://jaeger.dev.storj.io/trace/62ef902d4708d459
FAIL (2025-12-31 16:59:08 UTC)
orbiter warning: context deadline exceeded
Duration: 3.594151571s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X..........X.....................................X......X..] duration:
41.454237029s trace:
https://jaeger.dev.storj.io/trace/51209bc25994253c
FAIL (2025-12-31 16:59:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=64.49.13.169:4711 , piece_id=TN5KR3BMTLKVEXXQVISEDFLX6B3TSQV3FMY6TEMRXEYU5GR7Z2RA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.882880440 (625h27m34.753963475s) [noise=true , address=145.79.227.7:4314 , piece_id=6L5U7UKHUMYRHPAHZP7AIFDGHNNOIMBUTBD5AYMISGOMNQFUR7CA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.883312763 (625h27m34.754395808s) [noise=true , address=145.79.227.16:4931 , piece_id=MEKJOAWN564CXEFUGTUBYJNF2APXDVFV3UZGWESAKVFSVIPD6PTA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.883748182 (625h27m34.754831227s) [noise=true , address=77.90.31.10:4539 , piece_id=2UJELFLSKN6HTW6TPRVBGVYTVQAYQ64EXINWY2EDTRF62C2NYC6A, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.884108909 (625h27m34.755191944s) [noise=true , address=145.79.227.34:4939 , piece_id=4RO4RLYT2FJ2W5J2BXV5QBTRDBPEWXWTLWERVZGBXXZAPTF4MBKA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.884134819 (625h27m34.755217854s) [noise=true , address=77.90.31.12:4610 , piece_id=4W2K2M3RJ6X36C7XEBTWIGJWH34OINJJGUERDSIM5XNWDC4KTPDA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.884545450 (625h27m34.755628495s) [noise=true , address=45.91.116.6:4999 , piece_id=BNUMSSY7WN7S52XJT6PWA4X7B5PG5MR5WQYH2OYBYWMVLJGPZFZA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.884570638 (625h27m34.755653683s) [noise=true , address=77.90.31.16:4758 , piece_id=RQB3BR5PUIRIIUIPAH55MWGRAAFFPOKNMVHLYUX5VS7WSZVI5GJQ, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.884987812 (625h27m34.756070847s) [noise=true , address=77.90.31.10:4541 , piece_id=M3Y5NBJE7BBTZXFCIY3COZN7VEPSSDZA3AAIDAFKCFLSVVES2TLA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.885002600 (625h27m34.756085635s) [noise=true , address=45.91.116.12:4243 , piece_id=34VGEPVV2KHXFIR3X4VEUMX5UD4KJXZ5F573ODAMCJKF5ICIRTCQ, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.885364570 (625h27m34.756447615s) [noise=true , address=145.79.227.2:4119 , piece_id=UBSSEQ4EILR74PC22OSCWPQOFDKPY4FT74W3X7BOFIFI5OCI23MA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.885437358 (625h27m34.756520394s) [noise=true , address=77.90.31.17:4781 , piece_id=7U7FMPSPICGZNPYNQIGZA4DXQLUUHXAWV3NGIHTGLOFRGVV35DZA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.885713234 (625h27m34.756796269s) [noise=true , address=145.79.227.12:4793 , piece_id=NKCXNQYVEBULTP7UHHONXGF6ILTSUG6DZ2NCSCIFJHRYECDMVWFQ, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.886124126 (625h27m34.757207172s) [noise=true , address=145.79.227.49:4362 , piece_id=EXLZZCNKQIC22WCQVZD2G5EGQGEPI5OJLBFGVAESUANK2BP2JYAQ, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.886498589 (625h27m34.757581624s) [noise=true , address=145.79.227.29:4945 , piece_id=U367TT5ZN4GYR5FFCJCBZDS3FM3RZBWPGAMKFOPQSGHNZEYOTK3A, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.886844768 (625h27m34.757927803s) [noise=true , address=145.79.227.44:4213 , piece_id=6V767N4L5XZN655GRJWWTUUCNUPLEKVIW2PDEFAXUZZSFTTJQCTA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:36.347113954 (625h27m36.218196999s) [upload=548072]: Next returned nil segment
Dec 31 16:56:36.347127068 (625h27m36.218210114s) [upload=548072]: Waiting for error group managing segments...
Dec 31 16:56:36.782391943 (625h27m36.653474978s) [noise=true , address=145.79.227.2:4119 , piece_id=UBSSEQ4EILR74PC22OSCWPQOFDKPY4FT74W3X7BOFIFI5OCI23MA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:36.828828685 (625h27m36.69991172s) [noise=true , address=145.79.227.12:4793 , piece_id=NKCXNQYVEBULTP7UHHONXGF6ILTSUG6DZ2NCSCIFJHRYECDMVWFQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:36.909133386 (625h27m36.780216421s) [noise=true , address=145.79.227.7:4314 , piece_id=6L5U7UKHUMYRHPAHZP7AIFDGHNNOIMBUTBD5AYMISGOMNQFUR7CA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:36.951641147 (625h27m36.822724192s) [noise=true , address=77.90.31.10:4541 , piece_id=M3Y5NBJE7BBTZXFCIY3COZN7VEPSSDZA3AAIDAFKCFLSVVES2TLA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:36.953178845 (625h27m36.82426188s) [noise=true , address=77.90.31.16:4758 , piece_id=RQB3BR5PUIRIIUIPAH55MWGRAAFFPOKNMVHLYUX5VS7WSZVI5GJQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:36.967346992 (625h27m36.838430027s) [noise=true , address=77.90.31.10:4539 , piece_id=2UJELFLSKN6HTW6TPRVBGVYTVQAYQ64EXINWY2EDTRF62C2NYC6A, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.015751392 (625h27m36.886834427s) [noise=true , address=77.90.31.12:4610 , piece_id=4W2K2M3RJ6X36C7XEBTWIGJWH34OINJJGUERDSIM5XNWDC4KTPDA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.070458612 (625h27m36.941541647s) [noise=true , address=45.91.116.12:4243 , piece_id=34VGEPVV2KHXFIR3X4VEUMX5UD4KJXZ5F573ODAMCJKF5ICIRTCQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.277203082 (625h27m37.148286118s) [noise=true , address=77.90.31.17:4781 , piece_id=7U7FMPSPICGZNPYNQIGZA4DXQLUUHXAWV3NGIHTGLOFRGVV35DZA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.620497854 (625h27m37.491580909s) [noise=true , address=145.79.227.49:4362 , piece_id=EXLZZCNKQIC22WCQVZD2G5EGQGEPI5OJLBFGVAESUANK2BP2JYAQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.678713946 (625h27m37.549796991s) [noise=true , address=145.79.227.44:4213 , piece_id=6V767N4L5XZN655GRJWWTUUCNUPLEKVIW2PDEFAXUZZSFTTJQCTA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.686079791 (625h27m37.557162836s) [noise=true , address=145.79.227.33:4878 , piece_id=N5PTWWSHXTUK4LBAWQ5272RLIBLSCSFKML2DSCIVQCQN7NO7KFPQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.719337278 (625h27m37.590420313s) [noise=true , address=145.79.227.29:4945 , piece_id=U367TT5ZN4GYR5FFCJCBZDS3FM3RZBWPGAMKFOPQSGHNZEYOTK3A, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.738954522 (625h27m37.610037557s) [noise=true , address=145.79.227.34:4939 , piece_id=4RO4RLYT2FJ2W5J2BXV5QBTRDBPEWXWTLWERVZGBXXZAPTF4MBKA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.738974741 (625h27m37.610057776s) [seg_pos={0 0}, upload=548072]: Segment reached optimal threshold of 14 pieces.
Dec 31 16:56:37.739210258 (625h27m37.610293293s) [noise=true , address=145.79.227.16:4931 , piece_id=MEKJOAWN564CXEFUGTUBYJNF2APXDVFV3UZGWESAKVFSVIPD6PTA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rm4HueKsNdwL3eCTCSWoPseR2LQWcsE3jNPpxSfYQQorj9LL): protocol: expected piece hash; context canceled; EOF
Dec 31 16:56:37.739218525 (625h27m37.61030157s) [noise=true , address=145.79.227.16:4931 , piece_id=MEKJOAWN564CXEFUGTUBYJNF2APXDVFV3UZGWESAKVFSVIPD6PTA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold
Dec 31 16:56:37.739273609 (625h27m37.610356654s) [noise=true , address=193.149.164.143:4695 , piece_id=BAS4PIQCTI4ENCDY4BHGHNLQMYRZR2PW6PTNZJZAJ3X2U3JGOJIA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF
Dec 31 16:56:37.739292536 (625h27m37.610375581s) [noise=true , address=193.149.164.143:4695 , piece_id=BAS4PIQCTI4ENCDY4BHGHNLQMYRZR2PW6PTNZJZAJ3X2U3JGOJIA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold
Dec 31 16:56:37.739499520 (625h27m37.610582565s) [noise=true , address=45.91.116.6:4999 , piece_id=BNUMSSY7WN7S52XJT6PWA4X7B5PG5MR5WQYH2OYBYWMVLJGPZFZA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB): protocol: expected piece hash; context canceled; EOF
Dec 31 16:56:37.739506242 (625h27m37.610589277s) [noise=true , address=45.91.116.6:4999 , piece_id=BNUMSSY7WN7S52XJT6PWA4X7B5PG5MR5WQYH2OYBYWMVLJGPZFZA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold
Dec 31 16:56:37.739518896 (625h27m37.610601931s) [noise=true , address=64.49.13.169:4711 , piece_id=TN5KR3BMTLKVEXXQVISEDFLX6B3TSQV3FMY6TEMRXEYU5GR7Z2RA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60012->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60012->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60012->64.49.13.169:4711: use of closed network connection
Dec 31 16:56:37.739527362 (625h27m37.610610407s) [noise=true , address=64.49.13.169:4711 , piece_id=TN5KR3BMTLKVEXXQVISEDFLX6B3TSQV3FMY6TEMRXEYU5GR7Z2RA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold
Dec 31 16:56:37.739545647 (625h27m37.610628692s) [seg_pos={0 0}, upload=548072]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 16:56:37.739638374 (625h27m37.610721409s) [upload=548072]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:56:37.739643183 (625h27m37.610726228s) [upload=548072]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:56:38.030777232 (625h27m37.901860267s) [upload=548072]: Done uploading segments...
Dec 31 16:56:38.030797912 (625h27m37.901880947s) [upload=548072]: Upload finished. err: <nil>
Duration: 41.454253822s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [.......X....X.X...X....X.X.......X.XX..X..X...X.X.XX.X....X.] duration:
6.054595921s trace:
https://jaeger.dev.storj.io/trace/2670c1d470b783c0
FAIL (2025-12-31 16:58:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: XM72N5F4LM2GT7BFUB6KGTB7ZHBD3BKDIP2B3WQMZBRSXFTXYGUA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 6E2UBU3KP4CCJWTALFMI6OAWBWPUOPISX7KYHOKRA5LXC2TIZJRQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: F2UZ24G4HCUDZRFFP24SN2N2BBH7V63RWQ6PQ6ZQ476JI2GMDDAQ) context deadline exceeded
======LOGS======
Dec 31 16:57:47.565786811 (625h28m47.436869856s) [upload=548085]: Starting upload
Dec 31 16:57:47.565890649 (625h28m47.436973694s) [upload=548085]: Done starting upload
Dec 31 16:57:47.566112552 (625h28m47.437195597s) [upload=548085]: Uploading segments...
Dec 31 16:57:47.566156626 (625h28m47.437239681s) [upload=548085]: Got next segment. Inline: false
Dec 31 16:57:47.566185250 (625h28m47.437268296s) [upload=548085]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 16:57:47.566190200 (625h28m47.437273235s) [upload=548085]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 16:57:47.812075209 (625h28m47.683158244s) [seg_pos={0 0}, upload=548085]: Begin upload segment...
Dec 31 16:57:47.812130525 (625h28m47.68321356s) [seg_pos={0 0}, upload=548085]: Done begin upload segment.
Dec 31 16:57:47.812164800 (625h28m47.683247835s) [noise=true , address=145.79.227.13:4804 , piece_id=AAUFXV73JNGZBTMKBA3UNMVTVZMD6A3A6N5K7AU4TVSCMD3PXKUQ, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.812222480 (625h28m47.683305515s) [noise=true , address=45.91.116.12:4226 , piece_id=POOD3JH4PKGUPOKTW4CLUPDTRHAXP3MTQMRUDF2UB7SGBNE5KXHQ, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.813060045 (625h28m47.68414308s) [noise=true , address=145.79.227.27:4689 , piece_id=A44ST6SCYFVQENJFAVL6WDFOHLI5R2HMLYTJAI2B5XAECCF7YQJQ, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.813444006 (625h28m47.684527041s) [noise=true , address=145.79.227.44:4195 , piece_id=3KKGZV3EZHEBN6QA2XQ4GCSHZQYRBVT5PHIZCJQYGVCHH6BTNE6Q, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.813676348 (625h28m47.684759383s) [noise=true , address=145.79.227.5:4221 , piece_id=5XVHU4GOLP4BOEEOQ7YNLQ37LNCCBLIWIOU6BFVBROUH6X66MM7A, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.813843206 (625h28m47.684926241s) [noise=true , address=145.79.227.2:4113 , piece_id=GV72GLTUUXXGVYDMLRKV2QBXOE7IRN5MKCWJMN4KPCHIDJUV73NQ, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.814080999 (625h28m47.685164034s) [noise=true , address=77.90.31.10:4519 , piece_id=F2UZ24G4HCUDZRFFP24SN2N2BBH7V63RWQ6PQ6ZQ476JI2GMDDAQ, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.814381431 (625h28m47.685464476s) [noise=true , address=77.90.31.14:4661 , piece_id=MYADO6Y4UJOLKX63WSWWW27CM7W4TL6NQRIHXYF7RBWIF2XR3QWQ, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.814416498 (625h28m47.685499533s) [noise=true , address=145.79.227.18:4006 , piece_id=PCRVVFQ2WYWLWKSYCVV462GFGJLKWTXDOZM5N6NII6LT3F34YTRQ, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.814870462 (625h28m47.685953507s) [noise=true , address=145.79.227.22:4511 , piece_id=FIT2W6CM5Q77PSAWSPLOXDM2XD56NLDK5HLDEWY2A6NN3O4AQQAQ, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.814878157 (625h28m47.685961202s) [noise=true , address=145.79.227.9:4367 , piece_id=AOSHODY5MHZSG6EMAKEUDPKNDPZVULERG6NXVQWLSWHSC5UGO3IA, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.815238572 (625h28m47.686321608s) [noise=true , address=77.90.31.19:4841 , piece_id=XM72N5F4LM2GT7BFUB6KGTB7ZHBD3BKDIP2B3WQMZBRSXFTXYGUA, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.816120533 (625h28m47.687203578s) [noise=true , address=145.79.227.37:4567 , piece_id=EKKPOHUXXH437ZQ67K2LOPZ5773NJXLRIQTAKM7E7BTMRV5LLTJA, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.816236333 (625h28m47.687319368s) [noise=true , address=145.79.227.25:4619 , piece_id=MWXQL4ZDVJ2DO53MJQPPNO7EUFCQN6JYB5YSHQMKDBJRVEBRA3AA, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.817276784 (625h28m47.688359819s) [noise=true , address=45.91.116.8:4048 , piece_id=6E2UBU3KP4CCJWTALFMI6OAWBWPUOPISX7KYHOKRA5LXC2TIZJRQ, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.819322338 (625h28m47.690405373s) [noise=true , address=145.79.227.4:4187 , piece_id=53VLJWVLEWEFAVUKJVOGS6WCHUCA2APCUGRGA5VH4PDELLNU5EGA, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:49.512295273 (625h28m49.383378308s) [upload=548085]: Next returned nil segment
Dec 31 16:57:49.512308427 (625h28m49.383391462s) [upload=548085]: Waiting for error group managing segments...
Dec 31 16:57:49.787063029 (625h28m49.658146074s) [noise=true , address=145.79.227.13:4804 , piece_id=AAUFXV73JNGZBTMKBA3UNMVTVZMD6A3A6N5K7AU4TVSCMD3PXKUQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:49.794380723 (625h28m49.665463768s) [noise=true , address=145.79.227.2:4113 , piece_id=GV72GLTUUXXGVYDMLRKV2QBXOE7IRN5MKCWJMN4KPCHIDJUV73NQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:49.801155382 (625h28m49.672238417s) [noise=true , address=145.79.227.27:4689 , piece_id=A44ST6SCYFVQENJFAVL6WDFOHLI5R2HMLYTJAI2B5XAECCF7YQJQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:49.805611888 (625h28m49.676694923s) [noise=true , address=145.79.227.18:4006 , piece_id=PCRVVFQ2WYWLWKSYCVV462GFGJLKWTXDOZM5N6NII6LT3F34YTRQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:49.818638573 (625h28m49.689721608s) [noise=true , address=145.79.227.25:4619 , piece_id=MWXQL4ZDVJ2DO53MJQPPNO7EUFCQN6JYB5YSHQMKDBJRVEBRA3AA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:49.823609939 (625h28m49.694693005s) [noise=true , address=145.79.227.22:4511 , piece_id=FIT2W6CM5Q77PSAWSPLOXDM2XD56NLDK5HLDEWY2A6NN3O4AQQAQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:49.825557458 (625h28m49.696640493s) [noise=true , address=145.79.227.4:4187 , piece_id=53VLJWVLEWEFAVUKJVOGS6WCHUCA2APCUGRGA5VH4PDELLNU5EGA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:49.829627417 (625h28m49.700710452s) [noise=true , address=145.79.227.37:4567 , piece_id=EKKPOHUXXH437ZQ67K2LOPZ5773NJXLRIQTAKM7E7BTMRV5LLTJA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:49.829826126 (625h28m49.700909171s) [noise=true , address=145.79.227.9:4367 , piece_id=AOSHODY5MHZSG6EMAKEUDPKNDPZVULERG6NXVQWLSWHSC5UGO3IA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:49.972918433 (625h28m49.844001468s) [noise=true , address=145.79.227.5:4221 , piece_id=5XVHU4GOLP4BOEEOQ7YNLQ37LNCCBLIWIOU6BFVBROUH6X66MM7A, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:50.016337258 (625h28m49.887420303s) [noise=true , address=77.90.31.19:4841 , piece_id=XM72N5F4LM2GT7BFUB6KGTB7ZHBD3BKDIP2B3WQMZBRSXFTXYGUA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:50.066606578 (625h28m49.937689614s) [noise=true , address=77.90.31.10:4519 , piece_id=F2UZ24G4HCUDZRFFP24SN2N2BBH7V63RWQ6PQ6ZQ476JI2GMDDAQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:50.802894050 (625h28m50.673977085s) [noise=true , address=45.91.116.8:4048 , piece_id=6E2UBU3KP4CCJWTALFMI6OAWBWPUOPISX7KYHOKRA5LXC2TIZJRQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:50.802917805 (625h28m50.67400085s) [seg_pos={0 0}, upload=548085]: Segment reached optimal threshold of 13 pieces.
Dec 31 16:57:50.803136832 (625h28m50.674219877s) [noise=true , address=77.90.31.14:4661 , piece_id=MYADO6Y4UJOLKX63WSWWW27CM7W4TL6NQRIHXYF7RBWIF2XR3QWQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF
Dec 31 16:57:50.803144076 (625h28m50.674227121s) [noise=true , address=77.90.31.14:4661 , piece_id=MYADO6Y4UJOLKX63WSWWW27CM7W4TL6NQRIHXYF7RBWIF2XR3QWQ, seg_pos={0 0}, upload=548085]: Long tail cancellation due to optimal threshold
Dec 31 16:57:50.803253285 (625h28m50.67433633s) [noise=true , address=45.91.116.12:4226 , piece_id=POOD3JH4PKGUPOKTW4CLUPDTRHAXP3MTQMRUDF2UB7SGBNE5KXHQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF
Dec 31 16:57:50.803258725 (625h28m50.67434177s) [noise=true , address=45.91.116.12:4226 , piece_id=POOD3JH4PKGUPOKTW4CLUPDTRHAXP3MTQMRUDF2UB7SGBNE5KXHQ, seg_pos={0 0}, upload=548085]: Long tail cancellation due to optimal threshold
Dec 31 16:57:50.803484023 (625h28m50.674567068s) [noise=true , address=145.79.227.44:4195 , piece_id=3KKGZV3EZHEBN6QA2XQ4GCSHZQYRBVT5PHIZCJQYGVCHH6BTNE6Q, seg_pos={0 0}, upload=548085]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 31 16:57:50.803502799 (625h28m50.674585845s) [noise=true , address=145.79.227.44:4195 , piece_id=3KKGZV3EZHEBN6QA2XQ4GCSHZQYRBVT5PHIZCJQYGVCHH6BTNE6Q, seg_pos={0 0}, upload=548085]: Long tail cancellation due to optimal threshold
Dec 31 16:57:50.803532987 (625h28m50.674616032s) [seg_pos={0 0}, upload=548085]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 16:57:50.803658395 (625h28m50.67474145s) [upload=548085]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:57:50.803666380 (625h28m50.674749415s) [upload=548085]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:57:51.091543468 (625h28m50.962626503s) [upload=548085]: Done uploading segments...
Dec 31 16:57:51.091565551 (625h28m50.962648586s) [upload=548085]: Upload finished. err: <nil>
Duration: 6.054621941s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
36 out of 60 is failed (threshold=6) [.XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....X.X.XXX] duration:
5.001685104s trace:
https://jaeger.dev.storj.io/trace/62ef902d4708d459
FAIL (2025-12-31 16:58:49 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001709392s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X..........X.....................................X......X.] duration:
47.499427281s trace:
https://jaeger.dev.storj.io/trace/51209bc25994253c
FAIL (2025-12-31 16:58:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=64.49.13.169:4711 , piece_id=TN5KR3BMTLKVEXXQVISEDFLX6B3TSQV3FMY6TEMRXEYU5GR7Z2RA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.882880440 (625h27m34.753963475s) [noise=true , address=145.79.227.7:4314 , piece_id=6L5U7UKHUMYRHPAHZP7AIFDGHNNOIMBUTBD5AYMISGOMNQFUR7CA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.883312763 (625h27m34.754395808s) [noise=true , address=145.79.227.16:4931 , piece_id=MEKJOAWN564CXEFUGTUBYJNF2APXDVFV3UZGWESAKVFSVIPD6PTA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.883748182 (625h27m34.754831227s) [noise=true , address=77.90.31.10:4539 , piece_id=2UJELFLSKN6HTW6TPRVBGVYTVQAYQ64EXINWY2EDTRF62C2NYC6A, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.884108909 (625h27m34.755191944s) [noise=true , address=145.79.227.34:4939 , piece_id=4RO4RLYT2FJ2W5J2BXV5QBTRDBPEWXWTLWERVZGBXXZAPTF4MBKA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.884134819 (625h27m34.755217854s) [noise=true , address=77.90.31.12:4610 , piece_id=4W2K2M3RJ6X36C7XEBTWIGJWH34OINJJGUERDSIM5XNWDC4KTPDA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.884545450 (625h27m34.755628495s) [noise=true , address=45.91.116.6:4999 , piece_id=BNUMSSY7WN7S52XJT6PWA4X7B5PG5MR5WQYH2OYBYWMVLJGPZFZA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.884570638 (625h27m34.755653683s) [noise=true , address=77.90.31.16:4758 , piece_id=RQB3BR5PUIRIIUIPAH55MWGRAAFFPOKNMVHLYUX5VS7WSZVI5GJQ, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.884987812 (625h27m34.756070847s) [noise=true , address=77.90.31.10:4541 , piece_id=M3Y5NBJE7BBTZXFCIY3COZN7VEPSSDZA3AAIDAFKCFLSVVES2TLA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.885002600 (625h27m34.756085635s) [noise=true , address=45.91.116.12:4243 , piece_id=34VGEPVV2KHXFIR3X4VEUMX5UD4KJXZ5F573ODAMCJKF5ICIRTCQ, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.885364570 (625h27m34.756447615s) [noise=true , address=145.79.227.2:4119 , piece_id=UBSSEQ4EILR74PC22OSCWPQOFDKPY4FT74W3X7BOFIFI5OCI23MA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.885437358 (625h27m34.756520394s) [noise=true , address=77.90.31.17:4781 , piece_id=7U7FMPSPICGZNPYNQIGZA4DXQLUUHXAWV3NGIHTGLOFRGVV35DZA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.885713234 (625h27m34.756796269s) [noise=true , address=145.79.227.12:4793 , piece_id=NKCXNQYVEBULTP7UHHONXGF6ILTSUG6DZ2NCSCIFJHRYECDMVWFQ, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.886124126 (625h27m34.757207172s) [noise=true , address=145.79.227.49:4362 , piece_id=EXLZZCNKQIC22WCQVZD2G5EGQGEPI5OJLBFGVAESUANK2BP2JYAQ, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.886498589 (625h27m34.757581624s) [noise=true , address=145.79.227.29:4945 , piece_id=U367TT5ZN4GYR5FFCJCBZDS3FM3RZBWPGAMKFOPQSGHNZEYOTK3A, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.886844768 (625h27m34.757927803s) [noise=true , address=145.79.227.44:4213 , piece_id=6V767N4L5XZN655GRJWWTUUCNUPLEKVIW2PDEFAXUZZSFTTJQCTA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:36.347113954 (625h27m36.218196999s) [upload=548072]: Next returned nil segment
Dec 31 16:56:36.347127068 (625h27m36.218210114s) [upload=548072]: Waiting for error group managing segments...
Dec 31 16:56:36.782391943 (625h27m36.653474978s) [noise=true , address=145.79.227.2:4119 , piece_id=UBSSEQ4EILR74PC22OSCWPQOFDKPY4FT74W3X7BOFIFI5OCI23MA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:36.828828685 (625h27m36.69991172s) [noise=true , address=145.79.227.12:4793 , piece_id=NKCXNQYVEBULTP7UHHONXGF6ILTSUG6DZ2NCSCIFJHRYECDMVWFQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:36.909133386 (625h27m36.780216421s) [noise=true , address=145.79.227.7:4314 , piece_id=6L5U7UKHUMYRHPAHZP7AIFDGHNNOIMBUTBD5AYMISGOMNQFUR7CA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:36.951641147 (625h27m36.822724192s) [noise=true , address=77.90.31.10:4541 , piece_id=M3Y5NBJE7BBTZXFCIY3COZN7VEPSSDZA3AAIDAFKCFLSVVES2TLA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:36.953178845 (625h27m36.82426188s) [noise=true , address=77.90.31.16:4758 , piece_id=RQB3BR5PUIRIIUIPAH55MWGRAAFFPOKNMVHLYUX5VS7WSZVI5GJQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:36.967346992 (625h27m36.838430027s) [noise=true , address=77.90.31.10:4539 , piece_id=2UJELFLSKN6HTW6TPRVBGVYTVQAYQ64EXINWY2EDTRF62C2NYC6A, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.015751392 (625h27m36.886834427s) [noise=true , address=77.90.31.12:4610 , piece_id=4W2K2M3RJ6X36C7XEBTWIGJWH34OINJJGUERDSIM5XNWDC4KTPDA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.070458612 (625h27m36.941541647s) [noise=true , address=45.91.116.12:4243 , piece_id=34VGEPVV2KHXFIR3X4VEUMX5UD4KJXZ5F573ODAMCJKF5ICIRTCQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.277203082 (625h27m37.148286118s) [noise=true , address=77.90.31.17:4781 , piece_id=7U7FMPSPICGZNPYNQIGZA4DXQLUUHXAWV3NGIHTGLOFRGVV35DZA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.620497854 (625h27m37.491580909s) [noise=true , address=145.79.227.49:4362 , piece_id=EXLZZCNKQIC22WCQVZD2G5EGQGEPI5OJLBFGVAESUANK2BP2JYAQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.678713946 (625h27m37.549796991s) [noise=true , address=145.79.227.44:4213 , piece_id=6V767N4L5XZN655GRJWWTUUCNUPLEKVIW2PDEFAXUZZSFTTJQCTA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.686079791 (625h27m37.557162836s) [noise=true , address=145.79.227.33:4878 , piece_id=N5PTWWSHXTUK4LBAWQ5272RLIBLSCSFKML2DSCIVQCQN7NO7KFPQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.719337278 (625h27m37.590420313s) [noise=true , address=145.79.227.29:4945 , piece_id=U367TT5ZN4GYR5FFCJCBZDS3FM3RZBWPGAMKFOPQSGHNZEYOTK3A, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.738954522 (625h27m37.610037557s) [noise=true , address=145.79.227.34:4939 , piece_id=4RO4RLYT2FJ2W5J2BXV5QBTRDBPEWXWTLWERVZGBXXZAPTF4MBKA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.738974741 (625h27m37.610057776s) [seg_pos={0 0}, upload=548072]: Segment reached optimal threshold of 14 pieces.
Dec 31 16:56:37.739210258 (625h27m37.610293293s) [noise=true , address=145.79.227.16:4931 , piece_id=MEKJOAWN564CXEFUGTUBYJNF2APXDVFV3UZGWESAKVFSVIPD6PTA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rm4HueKsNdwL3eCTCSWoPseR2LQWcsE3jNPpxSfYQQorj9LL): protocol: expected piece hash; context canceled; EOF
Dec 31 16:56:37.739218525 (625h27m37.61030157s) [noise=true , address=145.79.227.16:4931 , piece_id=MEKJOAWN564CXEFUGTUBYJNF2APXDVFV3UZGWESAKVFSVIPD6PTA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold
Dec 31 16:56:37.739273609 (625h27m37.610356654s) [noise=true , address=193.149.164.143:4695 , piece_id=BAS4PIQCTI4ENCDY4BHGHNLQMYRZR2PW6PTNZJZAJ3X2U3JGOJIA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF
Dec 31 16:56:37.739292536 (625h27m37.610375581s) [noise=true , address=193.149.164.143:4695 , piece_id=BAS4PIQCTI4ENCDY4BHGHNLQMYRZR2PW6PTNZJZAJ3X2U3JGOJIA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold
Dec 31 16:56:37.739499520 (625h27m37.610582565s) [noise=true , address=45.91.116.6:4999 , piece_id=BNUMSSY7WN7S52XJT6PWA4X7B5PG5MR5WQYH2OYBYWMVLJGPZFZA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB): protocol: expected piece hash; context canceled; EOF
Dec 31 16:56:37.739506242 (625h27m37.610589277s) [noise=true , address=45.91.116.6:4999 , piece_id=BNUMSSY7WN7S52XJT6PWA4X7B5PG5MR5WQYH2OYBYWMVLJGPZFZA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold
Dec 31 16:56:37.739518896 (625h27m37.610601931s) [noise=true , address=64.49.13.169:4711 , piece_id=TN5KR3BMTLKVEXXQVISEDFLX6B3TSQV3FMY6TEMRXEYU5GR7Z2RA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60012->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60012->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60012->64.49.13.169:4711: use of closed network connection
Dec 31 16:56:37.739527362 (625h27m37.610610407s) [noise=true , address=64.49.13.169:4711 , piece_id=TN5KR3BMTLKVEXXQVISEDFLX6B3TSQV3FMY6TEMRXEYU5GR7Z2RA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold
Dec 31 16:56:37.739545647 (625h27m37.610628692s) [seg_pos={0 0}, upload=548072]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 16:56:37.739638374 (625h27m37.610721409s) [upload=548072]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:56:37.739643183 (625h27m37.610726228s) [upload=548072]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:56:38.030777232 (625h27m37.901860267s) [upload=548072]: Done uploading segments...
Dec 31 16:56:38.030797912 (625h27m37.901880947s) [upload=548072]: Upload finished. err: <nil>
Duration: 47.499442329s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [........X....X.X...X....X.X.......X.XX..X..X...X.X.XX.X....X] duration:
30.003628245s trace:
https://jaeger.dev.storj.io/trace/2670c1d470b783c0
FAIL (2025-12-31 16:58:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: XM72N5F4LM2GT7BFUB6KGTB7ZHBD3BKDIP2B3WQMZBRSXFTXYGUA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 6E2UBU3KP4CCJWTALFMI6OAWBWPUOPISX7KYHOKRA5LXC2TIZJRQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: F2UZ24G4HCUDZRFFP24SN2N2BBH7V63RWQ6PQ6ZQ476JI2GMDDAQ) context deadline exceeded
======LOGS======
Dec 31 16:57:47.565786811 (625h28m47.436869856s) [upload=548085]: Starting upload
Dec 31 16:57:47.565890649 (625h28m47.436973694s) [upload=548085]: Done starting upload
Dec 31 16:57:47.566112552 (625h28m47.437195597s) [upload=548085]: Uploading segments...
Dec 31 16:57:47.566156626 (625h28m47.437239681s) [upload=548085]: Got next segment. Inline: false
Dec 31 16:57:47.566185250 (625h28m47.437268296s) [upload=548085]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 16:57:47.566190200 (625h28m47.437273235s) [upload=548085]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 16:57:47.812075209 (625h28m47.683158244s) [seg_pos={0 0}, upload=548085]: Begin upload segment...
Dec 31 16:57:47.812130525 (625h28m47.68321356s) [seg_pos={0 0}, upload=548085]: Done begin upload segment.
Dec 31 16:57:47.812164800 (625h28m47.683247835s) [noise=true , address=145.79.227.13:4804 , piece_id=AAUFXV73JNGZBTMKBA3UNMVTVZMD6A3A6N5K7AU4TVSCMD3PXKUQ, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.812222480 (625h28m47.683305515s) [noise=true , address=45.91.116.12:4226 , piece_id=POOD3JH4PKGUPOKTW4CLUPDTRHAXP3MTQMRUDF2UB7SGBNE5KXHQ, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.813060045 (625h28m47.68414308s) [noise=true , address=145.79.227.27:4689 , piece_id=A44ST6SCYFVQENJFAVL6WDFOHLI5R2HMLYTJAI2B5XAECCF7YQJQ, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.813444006 (625h28m47.684527041s) [noise=true , address=145.79.227.44:4195 , piece_id=3KKGZV3EZHEBN6QA2XQ4GCSHZQYRBVT5PHIZCJQYGVCHH6BTNE6Q, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.813676348 (625h28m47.684759383s) [noise=true , address=145.79.227.5:4221 , piece_id=5XVHU4GOLP4BOEEOQ7YNLQ37LNCCBLIWIOU6BFVBROUH6X66MM7A, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.813843206 (625h28m47.684926241s) [noise=true , address=145.79.227.2:4113 , piece_id=GV72GLTUUXXGVYDMLRKV2QBXOE7IRN5MKCWJMN4KPCHIDJUV73NQ, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.814080999 (625h28m47.685164034s) [noise=true , address=77.90.31.10:4519 , piece_id=F2UZ24G4HCUDZRFFP24SN2N2BBH7V63RWQ6PQ6ZQ476JI2GMDDAQ, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.814381431 (625h28m47.685464476s) [noise=true , address=77.90.31.14:4661 , piece_id=MYADO6Y4UJOLKX63WSWWW27CM7W4TL6NQRIHXYF7RBWIF2XR3QWQ, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.814416498 (625h28m47.685499533s) [noise=true , address=145.79.227.18:4006 , piece_id=PCRVVFQ2WYWLWKSYCVV462GFGJLKWTXDOZM5N6NII6LT3F34YTRQ, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.814870462 (625h28m47.685953507s) [noise=true , address=145.79.227.22:4511 , piece_id=FIT2W6CM5Q77PSAWSPLOXDM2XD56NLDK5HLDEWY2A6NN3O4AQQAQ, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.814878157 (625h28m47.685961202s) [noise=true , address=145.79.227.9:4367 , piece_id=AOSHODY5MHZSG6EMAKEUDPKNDPZVULERG6NXVQWLSWHSC5UGO3IA, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.815238572 (625h28m47.686321608s) [noise=true , address=77.90.31.19:4841 , piece_id=XM72N5F4LM2GT7BFUB6KGTB7ZHBD3BKDIP2B3WQMZBRSXFTXYGUA, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.816120533 (625h28m47.687203578s) [noise=true , address=145.79.227.37:4567 , piece_id=EKKPOHUXXH437ZQ67K2LOPZ5773NJXLRIQTAKM7E7BTMRV5LLTJA, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.816236333 (625h28m47.687319368s) [noise=true , address=145.79.227.25:4619 , piece_id=MWXQL4ZDVJ2DO53MJQPPNO7EUFCQN6JYB5YSHQMKDBJRVEBRA3AA, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.817276784 (625h28m47.688359819s) [noise=true , address=45.91.116.8:4048 , piece_id=6E2UBU3KP4CCJWTALFMI6OAWBWPUOPISX7KYHOKRA5LXC2TIZJRQ, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:47.819322338 (625h28m47.690405373s) [noise=true , address=145.79.227.4:4187 , piece_id=53VLJWVLEWEFAVUKJVOGS6WCHUCA2APCUGRGA5VH4PDELLNU5EGA, seg_pos={0 0}, upload=548085]: Uploading piece...
Dec 31 16:57:49.512295273 (625h28m49.383378308s) [upload=548085]: Next returned nil segment
Dec 31 16:57:49.512308427 (625h28m49.383391462s) [upload=548085]: Waiting for error group managing segments...
Dec 31 16:57:49.787063029 (625h28m49.658146074s) [noise=true , address=145.79.227.13:4804 , piece_id=AAUFXV73JNGZBTMKBA3UNMVTVZMD6A3A6N5K7AU4TVSCMD3PXKUQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:49.794380723 (625h28m49.665463768s) [noise=true , address=145.79.227.2:4113 , piece_id=GV72GLTUUXXGVYDMLRKV2QBXOE7IRN5MKCWJMN4KPCHIDJUV73NQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:49.801155382 (625h28m49.672238417s) [noise=true , address=145.79.227.27:4689 , piece_id=A44ST6SCYFVQENJFAVL6WDFOHLI5R2HMLYTJAI2B5XAECCF7YQJQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:49.805611888 (625h28m49.676694923s) [noise=true , address=145.79.227.18:4006 , piece_id=PCRVVFQ2WYWLWKSYCVV462GFGJLKWTXDOZM5N6NII6LT3F34YTRQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:49.818638573 (625h28m49.689721608s) [noise=true , address=145.79.227.25:4619 , piece_id=MWXQL4ZDVJ2DO53MJQPPNO7EUFCQN6JYB5YSHQMKDBJRVEBRA3AA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:49.823609939 (625h28m49.694693005s) [noise=true , address=145.79.227.22:4511 , piece_id=FIT2W6CM5Q77PSAWSPLOXDM2XD56NLDK5HLDEWY2A6NN3O4AQQAQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:49.825557458 (625h28m49.696640493s) [noise=true , address=145.79.227.4:4187 , piece_id=53VLJWVLEWEFAVUKJVOGS6WCHUCA2APCUGRGA5VH4PDELLNU5EGA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:49.829627417 (625h28m49.700710452s) [noise=true , address=145.79.227.37:4567 , piece_id=EKKPOHUXXH437ZQ67K2LOPZ5773NJXLRIQTAKM7E7BTMRV5LLTJA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:49.829826126 (625h28m49.700909171s) [noise=true , address=145.79.227.9:4367 , piece_id=AOSHODY5MHZSG6EMAKEUDPKNDPZVULERG6NXVQWLSWHSC5UGO3IA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:49.972918433 (625h28m49.844001468s) [noise=true , address=145.79.227.5:4221 , piece_id=5XVHU4GOLP4BOEEOQ7YNLQ37LNCCBLIWIOU6BFVBROUH6X66MM7A, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:50.016337258 (625h28m49.887420303s) [noise=true , address=77.90.31.19:4841 , piece_id=XM72N5F4LM2GT7BFUB6KGTB7ZHBD3BKDIP2B3WQMZBRSXFTXYGUA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:50.066606578 (625h28m49.937689614s) [noise=true , address=77.90.31.10:4519 , piece_id=F2UZ24G4HCUDZRFFP24SN2N2BBH7V63RWQ6PQ6ZQ476JI2GMDDAQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:50.802894050 (625h28m50.673977085s) [noise=true , address=45.91.116.8:4048 , piece_id=6E2UBU3KP4CCJWTALFMI6OAWBWPUOPISX7KYHOKRA5LXC2TIZJRQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil>
Dec 31 16:57:50.802917805 (625h28m50.67400085s) [seg_pos={0 0}, upload=548085]: Segment reached optimal threshold of 13 pieces.
Dec 31 16:57:50.803136832 (625h28m50.674219877s) [noise=true , address=77.90.31.14:4661 , piece_id=MYADO6Y4UJOLKX63WSWWW27CM7W4TL6NQRIHXYF7RBWIF2XR3QWQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF
Dec 31 16:57:50.803144076 (625h28m50.674227121s) [noise=true , address=77.90.31.14:4661 , piece_id=MYADO6Y4UJOLKX63WSWWW27CM7W4TL6NQRIHXYF7RBWIF2XR3QWQ, seg_pos={0 0}, upload=548085]: Long tail cancellation due to optimal threshold
Dec 31 16:57:50.803253285 (625h28m50.67433633s) [noise=true , address=45.91.116.12:4226 , piece_id=POOD3JH4PKGUPOKTW4CLUPDTRHAXP3MTQMRUDF2UB7SGBNE5KXHQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF
Dec 31 16:57:50.803258725 (625h28m50.67434177s) [noise=true , address=45.91.116.12:4226 , piece_id=POOD3JH4PKGUPOKTW4CLUPDTRHAXP3MTQMRUDF2UB7SGBNE5KXHQ, seg_pos={0 0}, upload=548085]: Long tail cancellation due to optimal threshold
Dec 31 16:57:50.803484023 (625h28m50.674567068s) [noise=true , address=145.79.227.44:4195 , piece_id=3KKGZV3EZHEBN6QA2XQ4GCSHZQYRBVT5PHIZCJQYGVCHH6BTNE6Q, seg_pos={0 0}, upload=548085]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 31 16:57:50.803502799 (625h28m50.674585845s) [noise=true , address=145.79.227.44:4195 , piece_id=3KKGZV3EZHEBN6QA2XQ4GCSHZQYRBVT5PHIZCJQYGVCHH6BTNE6Q, seg_pos={0 0}, upload=548085]: Long tail cancellation due to optimal threshold
Dec 31 16:57:50.803532987 (625h28m50.674616032s) [seg_pos={0 0}, upload=548085]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 16:57:50.803658395 (625h28m50.67474145s) [upload=548085]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:57:50.803666380 (625h28m50.674749415s) [upload=548085]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:57:51.091543468 (625h28m50.962626503s) [upload=548085]: Done uploading segments...
Dec 31 16:57:51.091565551 (625h28m50.962648586s) [upload=548085]: Upload finished. err: <nil>
Duration: 30.003673311s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
35 out of 60 is failed (threshold=6) [..XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....X.X.XX] duration:
5.00075153s trace:
https://jaeger.dev.storj.io/trace/1165c148198ac14f
FAIL (2025-12-31 16:57:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.0007875s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X..........X.....................................X......X] duration:
1m0.004552763s trace:
https://jaeger.dev.storj.io/trace/51209bc25994253c
FAIL (2025-12-31 16:57:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=64.49.13.169:4711 , piece_id=TN5KR3BMTLKVEXXQVISEDFLX6B3TSQV3FMY6TEMRXEYU5GR7Z2RA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.882880440 (625h27m34.753963475s) [noise=true , address=145.79.227.7:4314 , piece_id=6L5U7UKHUMYRHPAHZP7AIFDGHNNOIMBUTBD5AYMISGOMNQFUR7CA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.883312763 (625h27m34.754395808s) [noise=true , address=145.79.227.16:4931 , piece_id=MEKJOAWN564CXEFUGTUBYJNF2APXDVFV3UZGWESAKVFSVIPD6PTA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.883748182 (625h27m34.754831227s) [noise=true , address=77.90.31.10:4539 , piece_id=2UJELFLSKN6HTW6TPRVBGVYTVQAYQ64EXINWY2EDTRF62C2NYC6A, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.884108909 (625h27m34.755191944s) [noise=true , address=145.79.227.34:4939 , piece_id=4RO4RLYT2FJ2W5J2BXV5QBTRDBPEWXWTLWERVZGBXXZAPTF4MBKA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.884134819 (625h27m34.755217854s) [noise=true , address=77.90.31.12:4610 , piece_id=4W2K2M3RJ6X36C7XEBTWIGJWH34OINJJGUERDSIM5XNWDC4KTPDA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.884545450 (625h27m34.755628495s) [noise=true , address=45.91.116.6:4999 , piece_id=BNUMSSY7WN7S52XJT6PWA4X7B5PG5MR5WQYH2OYBYWMVLJGPZFZA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.884570638 (625h27m34.755653683s) [noise=true , address=77.90.31.16:4758 , piece_id=RQB3BR5PUIRIIUIPAH55MWGRAAFFPOKNMVHLYUX5VS7WSZVI5GJQ, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.884987812 (625h27m34.756070847s) [noise=true , address=77.90.31.10:4541 , piece_id=M3Y5NBJE7BBTZXFCIY3COZN7VEPSSDZA3AAIDAFKCFLSVVES2TLA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.885002600 (625h27m34.756085635s) [noise=true , address=45.91.116.12:4243 , piece_id=34VGEPVV2KHXFIR3X4VEUMX5UD4KJXZ5F573ODAMCJKF5ICIRTCQ, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.885364570 (625h27m34.756447615s) [noise=true , address=145.79.227.2:4119 , piece_id=UBSSEQ4EILR74PC22OSCWPQOFDKPY4FT74W3X7BOFIFI5OCI23MA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.885437358 (625h27m34.756520394s) [noise=true , address=77.90.31.17:4781 , piece_id=7U7FMPSPICGZNPYNQIGZA4DXQLUUHXAWV3NGIHTGLOFRGVV35DZA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.885713234 (625h27m34.756796269s) [noise=true , address=145.79.227.12:4793 , piece_id=NKCXNQYVEBULTP7UHHONXGF6ILTSUG6DZ2NCSCIFJHRYECDMVWFQ, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.886124126 (625h27m34.757207172s) [noise=true , address=145.79.227.49:4362 , piece_id=EXLZZCNKQIC22WCQVZD2G5EGQGEPI5OJLBFGVAESUANK2BP2JYAQ, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.886498589 (625h27m34.757581624s) [noise=true , address=145.79.227.29:4945 , piece_id=U367TT5ZN4GYR5FFCJCBZDS3FM3RZBWPGAMKFOPQSGHNZEYOTK3A, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:34.886844768 (625h27m34.757927803s) [noise=true , address=145.79.227.44:4213 , piece_id=6V767N4L5XZN655GRJWWTUUCNUPLEKVIW2PDEFAXUZZSFTTJQCTA, seg_pos={0 0}, upload=548072]: Uploading piece...
Dec 31 16:56:36.347113954 (625h27m36.218196999s) [upload=548072]: Next returned nil segment
Dec 31 16:56:36.347127068 (625h27m36.218210114s) [upload=548072]: Waiting for error group managing segments...
Dec 31 16:56:36.782391943 (625h27m36.653474978s) [noise=true , address=145.79.227.2:4119 , piece_id=UBSSEQ4EILR74PC22OSCWPQOFDKPY4FT74W3X7BOFIFI5OCI23MA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:36.828828685 (625h27m36.69991172s) [noise=true , address=145.79.227.12:4793 , piece_id=NKCXNQYVEBULTP7UHHONXGF6ILTSUG6DZ2NCSCIFJHRYECDMVWFQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:36.909133386 (625h27m36.780216421s) [noise=true , address=145.79.227.7:4314 , piece_id=6L5U7UKHUMYRHPAHZP7AIFDGHNNOIMBUTBD5AYMISGOMNQFUR7CA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:36.951641147 (625h27m36.822724192s) [noise=true , address=77.90.31.10:4541 , piece_id=M3Y5NBJE7BBTZXFCIY3COZN7VEPSSDZA3AAIDAFKCFLSVVES2TLA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:36.953178845 (625h27m36.82426188s) [noise=true , address=77.90.31.16:4758 , piece_id=RQB3BR5PUIRIIUIPAH55MWGRAAFFPOKNMVHLYUX5VS7WSZVI5GJQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:36.967346992 (625h27m36.838430027s) [noise=true , address=77.90.31.10:4539 , piece_id=2UJELFLSKN6HTW6TPRVBGVYTVQAYQ64EXINWY2EDTRF62C2NYC6A, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.015751392 (625h27m36.886834427s) [noise=true , address=77.90.31.12:4610 , piece_id=4W2K2M3RJ6X36C7XEBTWIGJWH34OINJJGUERDSIM5XNWDC4KTPDA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.070458612 (625h27m36.941541647s) [noise=true , address=45.91.116.12:4243 , piece_id=34VGEPVV2KHXFIR3X4VEUMX5UD4KJXZ5F573ODAMCJKF5ICIRTCQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.277203082 (625h27m37.148286118s) [noise=true , address=77.90.31.17:4781 , piece_id=7U7FMPSPICGZNPYNQIGZA4DXQLUUHXAWV3NGIHTGLOFRGVV35DZA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.620497854 (625h27m37.491580909s) [noise=true , address=145.79.227.49:4362 , piece_id=EXLZZCNKQIC22WCQVZD2G5EGQGEPI5OJLBFGVAESUANK2BP2JYAQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.678713946 (625h27m37.549796991s) [noise=true , address=145.79.227.44:4213 , piece_id=6V767N4L5XZN655GRJWWTUUCNUPLEKVIW2PDEFAXUZZSFTTJQCTA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.686079791 (625h27m37.557162836s) [noise=true , address=145.79.227.33:4878 , piece_id=N5PTWWSHXTUK4LBAWQ5272RLIBLSCSFKML2DSCIVQCQN7NO7KFPQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.719337278 (625h27m37.590420313s) [noise=true , address=145.79.227.29:4945 , piece_id=U367TT5ZN4GYR5FFCJCBZDS3FM3RZBWPGAMKFOPQSGHNZEYOTK3A, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.738954522 (625h27m37.610037557s) [noise=true , address=145.79.227.34:4939 , piece_id=4RO4RLYT2FJ2W5J2BXV5QBTRDBPEWXWTLWERVZGBXXZAPTF4MBKA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil>
Dec 31 16:56:37.738974741 (625h27m37.610057776s) [seg_pos={0 0}, upload=548072]: Segment reached optimal threshold of 14 pieces.
Dec 31 16:56:37.739210258 (625h27m37.610293293s) [noise=true , address=145.79.227.16:4931 , piece_id=MEKJOAWN564CXEFUGTUBYJNF2APXDVFV3UZGWESAKVFSVIPD6PTA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rm4HueKsNdwL3eCTCSWoPseR2LQWcsE3jNPpxSfYQQorj9LL): protocol: expected piece hash; context canceled; EOF
Dec 31 16:56:37.739218525 (625h27m37.61030157s) [noise=true , address=145.79.227.16:4931 , piece_id=MEKJOAWN564CXEFUGTUBYJNF2APXDVFV3UZGWESAKVFSVIPD6PTA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold
Dec 31 16:56:37.739273609 (625h27m37.610356654s) [noise=true , address=193.149.164.143:4695 , piece_id=BAS4PIQCTI4ENCDY4BHGHNLQMYRZR2PW6PTNZJZAJ3X2U3JGOJIA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF
Dec 31 16:56:37.739292536 (625h27m37.610375581s) [noise=true , address=193.149.164.143:4695 , piece_id=BAS4PIQCTI4ENCDY4BHGHNLQMYRZR2PW6PTNZJZAJ3X2U3JGOJIA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold
Dec 31 16:56:37.739499520 (625h27m37.610582565s) [noise=true , address=45.91.116.6:4999 , piece_id=BNUMSSY7WN7S52XJT6PWA4X7B5PG5MR5WQYH2OYBYWMVLJGPZFZA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB): protocol: expected piece hash; context canceled; EOF
Dec 31 16:56:37.739506242 (625h27m37.610589277s) [noise=true , address=45.91.116.6:4999 , piece_id=BNUMSSY7WN7S52XJT6PWA4X7B5PG5MR5WQYH2OYBYWMVLJGPZFZA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold
Dec 31 16:56:37.739518896 (625h27m37.610601931s) [noise=true , address=64.49.13.169:4711 , piece_id=TN5KR3BMTLKVEXXQVISEDFLX6B3TSQV3FMY6TEMRXEYU5GR7Z2RA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60012->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60012->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60012->64.49.13.169:4711: use of closed network connection
Dec 31 16:56:37.739527362 (625h27m37.610610407s) [noise=true , address=64.49.13.169:4711 , piece_id=TN5KR3BMTLKVEXXQVISEDFLX6B3TSQV3FMY6TEMRXEYU5GR7Z2RA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold
Dec 31 16:56:37.739545647 (625h27m37.610628692s) [seg_pos={0 0}, upload=548072]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 16:56:37.739638374 (625h27m37.610721409s) [upload=548072]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:56:37.739643183 (625h27m37.610726228s) [upload=548072]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:56:38.030777232 (625h27m37.901860267s) [upload=548072]: Done uploading segments...
Dec 31 16:56:38.030797912 (625h27m37.901880947s) [upload=548072]: Upload finished. err: <nil>
Duration: 1m0.004563663s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
34 out of 60 is failed (threshold=6) [...XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....X.X.X] duration:
5.000918597s trace:
https://jaeger.dev.storj.io/trace/15df357c2a777d6
FAIL (2025-12-31 16:57:17 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000943464s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [X........X....X.X...X....X.X.......X.XX..X..X...X.X.XX.X....] duration:
6.25788509s trace:
https://jaeger.dev.storj.io/trace/2357ddb7e2c2e281
FAIL (2025-12-31 16:56:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 16:54:04.303380506 (625h25m4.174463542s) [upload=548032]: Starting upload
Dec 31 16:54:04.303432836 (625h25m4.174515871s) [upload=548032]: Done starting upload
Dec 31 16:54:04.303545921 (625h25m4.174628976s) [upload=548032]: Uploading segments...
Dec 31 16:54:04.303631374 (625h25m4.174714449s) [upload=548032]: Got next segment. Inline: false
Dec 31 16:54:04.303679495 (625h25m4.17476254s) [upload=548032]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 16:54:04.303700816 (625h25m4.174783851s) [upload=548032]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 16:54:04.550160820 (625h25m4.421243865s) [seg_pos={0 0}, upload=548032]: Begin upload segment...
Dec 31 16:54:04.550206296 (625h25m4.421289341s) [seg_pos={0 0}, upload=548032]: Done begin upload segment.
Dec 31 16:54:04.550233237 (625h25m4.421316272s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.550319773 (625h25m4.421402848s) [noise=true , address=145.79.227.15:4874 , piece_id=3CVWADJWSRXY6B5SR5NCXK2ROZKQDUWX66NPHV3B3HN3AC72666Q, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.550895328 (625h25m4.421978363s) [noise=true , address=77.90.31.14:4662 , piece_id=Y3JXTJ3RYGB2JNDSRGS4XDA5BOVOS6WEGK44HZLYYLDTPOQXXQTA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.551320608 (625h25m4.422403653s) [noise=true , address=45.91.116.13:4153 , piece_id=MNIXKBZDEUHYK2Q66CVJFJJDNJMRICPWD4QVUZNKBOVSMKXIW4EA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.551783460 (625h25m4.422866495s) [noise=true , address=145.79.227.19:4079 , piece_id=TYQFG66ZLQRS5OTMLNOYRR24B7HF6FJKEBOZNZK2EO4I3PIU65AQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.552138956 (625h25m4.423221992s) [noise=true , address=145.79.227.38:4977 , piece_id=LZHY2WNZ6X23TN65R4PVJWLDYV4K57BQ4C6LRPXHVYNYIEO57S7Q, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.552418960 (625h25m4.423502005s) [noise=true , address=145.79.227.4:4185 , piece_id=RI6CLWZ2ZXPQ36SWRQTRI3ECAFTSHUK5TWFKYYXTE7CERSD3YGIQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.552838599 (625h25m4.423921644s) [noise=true , address=45.91.116.8:4046 , piece_id=YKXTIPM2QQD45UADUYGAUXH4TQFJII3SUKJ3W5NMYJYLNF4O57TA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.554332483 (625h25m4.425415528s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.554476558 (625h25m4.425559593s) [noise=true , address=77.90.31.10:4517 , piece_id=NDSDOK2KXTFHPZ237OIQOEKEWR5PAEJKN6L4RFXSYNHFXISGYHPA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.555638550 (625h25m4.426721585s) [noise=true , address=145.79.227.9:4367 , piece_id=MGWDE3UGN7OTFEHKEZKF3CTD4WMZWOXYIKU7543ZXQAXHHBQTI7Q, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.556990394 (625h25m4.428073439s) [noise=true , address=77.90.31.15:4697 , piece_id=MNW477DNTP25EHJMHVVW3BX3UYDBJGG7FXHFSTFZLTM2ZW4NV4ZQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.565033940 (625h25m4.436116975s) [noise=true , address=45.91.116.14:4191 , piece_id=HSXBPP3II2CHJHX3BFXR4BNCTAQDGLIRGBAPGHV7ZUP5IDMBTDQA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.565184408 (625h25m4.436267443s) [noise=true , address=145.79.227.14:4837 , piece_id=JHFAKK2Q6GUN3LSB7OLZE4QH5BQ3V2ANPT4QG6FQ7XC3SYPXSFLQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.566499170 (625h25m4.437582215s) [noise=true , address=145.79.227.52:4445 , piece_id=KLVNKKAVLTCDXBPGQDFLEOXOODU2KS4MPO4Y6OEPSVMX4AXRRWEQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.567394656 (625h25m4.438477701s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:06.264153690 (625h25m6.135236735s) [upload=548032]: Next returned nil segment
Dec 31 16:54:06.264177514 (625h25m6.135260559s) [upload=548032]: Waiting for error group managing segments...
Dec 31 16:54:06.536331012 (625h25m6.407414047s) [noise=true , address=145.79.227.52:4445 , piece_id=KLVNKKAVLTCDXBPGQDFLEOXOODU2KS4MPO4Y6OEPSVMX4AXRRWEQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.538792870 (625h25m6.409875906s) [noise=true , address=145.79.227.15:4874 , piece_id=3CVWADJWSRXY6B5SR5NCXK2ROZKQDUWX66NPHV3B3HN3AC72666Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.559727663 (625h25m6.430810698s) [noise=true , address=145.79.227.4:4185 , piece_id=RI6CLWZ2ZXPQ36SWRQTRI3ECAFTSHUK5TWFKYYXTE7CERSD3YGIQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.560250448 (625h25m6.431333493s) [noise=true , address=145.79.227.9:4367 , piece_id=MGWDE3UGN7OTFEHKEZKF3CTD4WMZWOXYIKU7543ZXQAXHHBQTI7Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.569094128 (625h25m6.440177173s) [noise=true , address=145.79.227.14:4837 , piece_id=JHFAKK2Q6GUN3LSB7OLZE4QH5BQ3V2ANPT4QG6FQ7XC3SYPXSFLQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.626771975 (625h25m6.49785501s) [noise=true , address=145.79.227.38:4977 , piece_id=LZHY2WNZ6X23TN65R4PVJWLDYV4K57BQ4C6LRPXHVYNYIEO57S7Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.636667477 (625h25m6.507750522s) [noise=true , address=145.79.227.19:4079 , piece_id=TYQFG66ZLQRS5OTMLNOYRR24B7HF6FJKEBOZNZK2EO4I3PIU65AQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.741721214 (625h25m6.612804259s) [noise=true , address=77.90.31.10:4517 , piece_id=NDSDOK2KXTFHPZ237OIQOEKEWR5PAEJKN6L4RFXSYNHFXISGYHPA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.863029281 (625h25m6.734112316s) [noise=true , address=45.91.116.13:4153 , piece_id=MNIXKBZDEUHYK2Q66CVJFJJDNJMRICPWD4QVUZNKBOVSMKXIW4EA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.904267314 (625h25m6.775350349s) [noise=true , address=77.90.31.15:4697 , piece_id=MNW477DNTP25EHJMHVVW3BX3UYDBJGG7FXHFSTFZLTM2ZW4NV4ZQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.999818748 (625h25m6.870901783s) [noise=true , address=45.91.116.14:4191 , piece_id=HSXBPP3II2CHJHX3BFXR4BNCTAQDGLIRGBAPGHV7ZUP5IDMBTDQA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:07.503007990 (625h25m7.374091025s) [noise=true , address=77.90.31.14:4662 , piece_id=Y3JXTJ3RYGB2JNDSRGS4XDA5BOVOS6WEGK44HZLYYLDTPOQXXQTA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:07.956546069 (625h25m7.827629114s) [noise=true , address=45.91.116.8:4046 , piece_id=YKXTIPM2QQD45UADUYGAUXH4TQFJII3SUKJ3W5NMYJYLNF4O57TA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:07.956571196 (625h25m7.827654231s) [seg_pos={0 0}, upload=548032]: Segment reached optimal threshold of 13 pieces.
Dec 31 16:54:07.956812125 (625h25m7.82789517s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF
Dec 31 16:54:07.956823177 (625h25m7.827906222s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold
Dec 31 16:54:07.957133406 (625h25m7.828216442s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF
Dec 31 16:54:07.957148957 (625h25m7.828231992s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold
Dec 31 16:54:07.957231634 (625h25m7.828314689s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF
Dec 31 16:54:07.957242886 (625h25m7.828325931s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold
Dec 31 16:54:07.957268374 (625h25m7.828351409s) [seg_pos={0 0}, upload=548032]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 16:54:07.957349389 (625h25m7.828432434s) [upload=548032]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:54:07.957356853 (625h25m7.828439898s) [upload=548032]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:54:08.248058148 (625h25m8.119141213s) [upload=548032]: Done uploading segments...
Dec 31 16:54:08.248084378 (625h25m8.119167413s) [upload=548032]: Upload finished. err: <nil>
Duration: 6.257910188s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [....X..........X.....................................X......] duration:
33.863640203s trace:
https://jaeger.dev.storj.io/trace/1fe9f44dea5d609c
FAIL (2025-12-31 16:56:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
.241246563 (625h22m16.112329598s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.241981543 (625h22m16.113064588s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242111701 (625h22m16.113194746s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242433975 (625h22m16.11351702s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242713457 (625h22m16.113796492s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242802286 (625h22m16.113885331s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.243245350 (625h22m16.114328395s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.243631865 (625h22m16.114714901s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.244008643 (625h22m16.115091678s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.244381102 (625h22m16.115464147s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.244758581 (625h22m16.115841616s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.245176256 (625h22m16.116259301s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.245538285 (625h22m16.11662133s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.245960649 (625h22m16.117043684s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.246337136 (625h22m16.117420181s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.247466537 (625h22m16.118549572s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.247996927 (625h22m16.119079962s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:17.693671241 (625h22m17.564754286s) [upload=547992]: Next returned nil segment
Dec 31 16:51:17.693714864 (625h22m17.564797899s) [upload=547992]: Waiting for error group managing segments...
Dec 31 16:51:18.120111632 (625h22m17.991194667s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.122023212 (625h22m17.993106257s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.201357746 (625h22m18.072440781s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.293128900 (625h22m18.164211965s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.306933417 (625h22m18.178016452s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.357131932 (625h22m18.228214977s) [noise=true , address=77.90.31.11:4563 , piece_id=2LRC66FCB4DTXU44NFN5O5L6ZMGDCQ6LHHB7TLYJ2LZSYKRWJEYQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.390427070 (625h22m18.261510105s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.469358907 (625h22m18.340441942s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.675586202 (625h22m18.546669267s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.834344079 (625h22m18.705427114s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.845713768 (625h22m18.716796813s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.980140516 (625h22m18.851223551s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.980522623 (625h22m18.851605658s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.997884583 (625h22m18.868967618s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.997904631 (625h22m18.868987666s) [seg_pos={0 0}, upload=547992]: Segment reached optimal threshold of 14 pieces.
Dec 31 16:51:18.998092839 (625h22m18.869175874s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998099372 (625h22m18.869182417s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998204202 (625h22m18.869287237s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998210183 (625h22m18.869293218s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998213039 (625h22m18.869296074s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998221765 (625h22m18.86930481s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998312208 (625h22m18.869395243s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122S3PRcJwYfK5UEWESnYA7aTPHA6zCLFyBuFSm2Pb5wFSxDPiR): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998318149 (625h22m18.869401184s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998356041 (625h22m18.869439076s) [seg_pos={0 0}, upload=547992]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 16:51:18.998428418 (625h22m18.869511454s) [upload=547992]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:51:18.998433228 (625h22m18.869516273s) [upload=547992]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:51:19.285646870 (625h22m19.156729935s) [upload=547992]: Done uploading segments...
Dec 31 16:51:19.285688400 (625h22m19.156771445s) [upload=547992]: Upload finished. err: <nil>
Duration: 33.863672965s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [....XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....X.X.] duration:
3.755160296s trace:
https://jaeger.dev.storj.io/trace/5e25e590cc565f32
FAIL (2025-12-31 16:56:31 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.755184471s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [XX........X....X.X...X....X.X.......X.XX..X..X...X.X.XX.X...] duration:
5.965832814s trace:
https://jaeger.dev.storj.io/trace/2357ddb7e2c2e281
FAIL (2025-12-31 16:56:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 16:54:04.303380506 (625h25m4.174463542s) [upload=548032]: Starting upload
Dec 31 16:54:04.303432836 (625h25m4.174515871s) [upload=548032]: Done starting upload
Dec 31 16:54:04.303545921 (625h25m4.174628976s) [upload=548032]: Uploading segments...
Dec 31 16:54:04.303631374 (625h25m4.174714449s) [upload=548032]: Got next segment. Inline: false
Dec 31 16:54:04.303679495 (625h25m4.17476254s) [upload=548032]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 16:54:04.303700816 (625h25m4.174783851s) [upload=548032]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 16:54:04.550160820 (625h25m4.421243865s) [seg_pos={0 0}, upload=548032]: Begin upload segment...
Dec 31 16:54:04.550206296 (625h25m4.421289341s) [seg_pos={0 0}, upload=548032]: Done begin upload segment.
Dec 31 16:54:04.550233237 (625h25m4.421316272s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.550319773 (625h25m4.421402848s) [noise=true , address=145.79.227.15:4874 , piece_id=3CVWADJWSRXY6B5SR5NCXK2ROZKQDUWX66NPHV3B3HN3AC72666Q, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.550895328 (625h25m4.421978363s) [noise=true , address=77.90.31.14:4662 , piece_id=Y3JXTJ3RYGB2JNDSRGS4XDA5BOVOS6WEGK44HZLYYLDTPOQXXQTA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.551320608 (625h25m4.422403653s) [noise=true , address=45.91.116.13:4153 , piece_id=MNIXKBZDEUHYK2Q66CVJFJJDNJMRICPWD4QVUZNKBOVSMKXIW4EA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.551783460 (625h25m4.422866495s) [noise=true , address=145.79.227.19:4079 , piece_id=TYQFG66ZLQRS5OTMLNOYRR24B7HF6FJKEBOZNZK2EO4I3PIU65AQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.552138956 (625h25m4.423221992s) [noise=true , address=145.79.227.38:4977 , piece_id=LZHY2WNZ6X23TN65R4PVJWLDYV4K57BQ4C6LRPXHVYNYIEO57S7Q, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.552418960 (625h25m4.423502005s) [noise=true , address=145.79.227.4:4185 , piece_id=RI6CLWZ2ZXPQ36SWRQTRI3ECAFTSHUK5TWFKYYXTE7CERSD3YGIQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.552838599 (625h25m4.423921644s) [noise=true , address=45.91.116.8:4046 , piece_id=YKXTIPM2QQD45UADUYGAUXH4TQFJII3SUKJ3W5NMYJYLNF4O57TA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.554332483 (625h25m4.425415528s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.554476558 (625h25m4.425559593s) [noise=true , address=77.90.31.10:4517 , piece_id=NDSDOK2KXTFHPZ237OIQOEKEWR5PAEJKN6L4RFXSYNHFXISGYHPA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.555638550 (625h25m4.426721585s) [noise=true , address=145.79.227.9:4367 , piece_id=MGWDE3UGN7OTFEHKEZKF3CTD4WMZWOXYIKU7543ZXQAXHHBQTI7Q, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.556990394 (625h25m4.428073439s) [noise=true , address=77.90.31.15:4697 , piece_id=MNW477DNTP25EHJMHVVW3BX3UYDBJGG7FXHFSTFZLTM2ZW4NV4ZQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.565033940 (625h25m4.436116975s) [noise=true , address=45.91.116.14:4191 , piece_id=HSXBPP3II2CHJHX3BFXR4BNCTAQDGLIRGBAPGHV7ZUP5IDMBTDQA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.565184408 (625h25m4.436267443s) [noise=true , address=145.79.227.14:4837 , piece_id=JHFAKK2Q6GUN3LSB7OLZE4QH5BQ3V2ANPT4QG6FQ7XC3SYPXSFLQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.566499170 (625h25m4.437582215s) [noise=true , address=145.79.227.52:4445 , piece_id=KLVNKKAVLTCDXBPGQDFLEOXOODU2KS4MPO4Y6OEPSVMX4AXRRWEQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.567394656 (625h25m4.438477701s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:06.264153690 (625h25m6.135236735s) [upload=548032]: Next returned nil segment
Dec 31 16:54:06.264177514 (625h25m6.135260559s) [upload=548032]: Waiting for error group managing segments...
Dec 31 16:54:06.536331012 (625h25m6.407414047s) [noise=true , address=145.79.227.52:4445 , piece_id=KLVNKKAVLTCDXBPGQDFLEOXOODU2KS4MPO4Y6OEPSVMX4AXRRWEQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.538792870 (625h25m6.409875906s) [noise=true , address=145.79.227.15:4874 , piece_id=3CVWADJWSRXY6B5SR5NCXK2ROZKQDUWX66NPHV3B3HN3AC72666Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.559727663 (625h25m6.430810698s) [noise=true , address=145.79.227.4:4185 , piece_id=RI6CLWZ2ZXPQ36SWRQTRI3ECAFTSHUK5TWFKYYXTE7CERSD3YGIQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.560250448 (625h25m6.431333493s) [noise=true , address=145.79.227.9:4367 , piece_id=MGWDE3UGN7OTFEHKEZKF3CTD4WMZWOXYIKU7543ZXQAXHHBQTI7Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.569094128 (625h25m6.440177173s) [noise=true , address=145.79.227.14:4837 , piece_id=JHFAKK2Q6GUN3LSB7OLZE4QH5BQ3V2ANPT4QG6FQ7XC3SYPXSFLQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.626771975 (625h25m6.49785501s) [noise=true , address=145.79.227.38:4977 , piece_id=LZHY2WNZ6X23TN65R4PVJWLDYV4K57BQ4C6LRPXHVYNYIEO57S7Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.636667477 (625h25m6.507750522s) [noise=true , address=145.79.227.19:4079 , piece_id=TYQFG66ZLQRS5OTMLNOYRR24B7HF6FJKEBOZNZK2EO4I3PIU65AQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.741721214 (625h25m6.612804259s) [noise=true , address=77.90.31.10:4517 , piece_id=NDSDOK2KXTFHPZ237OIQOEKEWR5PAEJKN6L4RFXSYNHFXISGYHPA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.863029281 (625h25m6.734112316s) [noise=true , address=45.91.116.13:4153 , piece_id=MNIXKBZDEUHYK2Q66CVJFJJDNJMRICPWD4QVUZNKBOVSMKXIW4EA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.904267314 (625h25m6.775350349s) [noise=true , address=77.90.31.15:4697 , piece_id=MNW477DNTP25EHJMHVVW3BX3UYDBJGG7FXHFSTFZLTM2ZW4NV4ZQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.999818748 (625h25m6.870901783s) [noise=true , address=45.91.116.14:4191 , piece_id=HSXBPP3II2CHJHX3BFXR4BNCTAQDGLIRGBAPGHV7ZUP5IDMBTDQA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:07.503007990 (625h25m7.374091025s) [noise=true , address=77.90.31.14:4662 , piece_id=Y3JXTJ3RYGB2JNDSRGS4XDA5BOVOS6WEGK44HZLYYLDTPOQXXQTA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:07.956546069 (625h25m7.827629114s) [noise=true , address=45.91.116.8:4046 , piece_id=YKXTIPM2QQD45UADUYGAUXH4TQFJII3SUKJ3W5NMYJYLNF4O57TA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:07.956571196 (625h25m7.827654231s) [seg_pos={0 0}, upload=548032]: Segment reached optimal threshold of 13 pieces.
Dec 31 16:54:07.956812125 (625h25m7.82789517s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF
Dec 31 16:54:07.956823177 (625h25m7.827906222s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold
Dec 31 16:54:07.957133406 (625h25m7.828216442s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF
Dec 31 16:54:07.957148957 (625h25m7.828231992s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold
Dec 31 16:54:07.957231634 (625h25m7.828314689s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF
Dec 31 16:54:07.957242886 (625h25m7.828325931s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold
Dec 31 16:54:07.957268374 (625h25m7.828351409s) [seg_pos={0 0}, upload=548032]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 16:54:07.957349389 (625h25m7.828432434s) [upload=548032]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:54:07.957356853 (625h25m7.828439898s) [upload=548032]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:54:08.248058148 (625h25m8.119141213s) [upload=548032]: Done uploading segments...
Dec 31 16:54:08.248084378 (625h25m8.119167413s) [upload=548032]: Upload finished. err: <nil>
Duration: 5.965877629s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.XX........X....X.X...X....X.X.......X.XX..X..X...X.X.XX.X..] duration:
6.073074422s trace:
https://jaeger.dev.storj.io/trace/2357ddb7e2c2e281
FAIL (2025-12-31 16:55:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 16:54:04.303380506 (625h25m4.174463542s) [upload=548032]: Starting upload
Dec 31 16:54:04.303432836 (625h25m4.174515871s) [upload=548032]: Done starting upload
Dec 31 16:54:04.303545921 (625h25m4.174628976s) [upload=548032]: Uploading segments...
Dec 31 16:54:04.303631374 (625h25m4.174714449s) [upload=548032]: Got next segment. Inline: false
Dec 31 16:54:04.303679495 (625h25m4.17476254s) [upload=548032]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 16:54:04.303700816 (625h25m4.174783851s) [upload=548032]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 16:54:04.550160820 (625h25m4.421243865s) [seg_pos={0 0}, upload=548032]: Begin upload segment...
Dec 31 16:54:04.550206296 (625h25m4.421289341s) [seg_pos={0 0}, upload=548032]: Done begin upload segment.
Dec 31 16:54:04.550233237 (625h25m4.421316272s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.550319773 (625h25m4.421402848s) [noise=true , address=145.79.227.15:4874 , piece_id=3CVWADJWSRXY6B5SR5NCXK2ROZKQDUWX66NPHV3B3HN3AC72666Q, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.550895328 (625h25m4.421978363s) [noise=true , address=77.90.31.14:4662 , piece_id=Y3JXTJ3RYGB2JNDSRGS4XDA5BOVOS6WEGK44HZLYYLDTPOQXXQTA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.551320608 (625h25m4.422403653s) [noise=true , address=45.91.116.13:4153 , piece_id=MNIXKBZDEUHYK2Q66CVJFJJDNJMRICPWD4QVUZNKBOVSMKXIW4EA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.551783460 (625h25m4.422866495s) [noise=true , address=145.79.227.19:4079 , piece_id=TYQFG66ZLQRS5OTMLNOYRR24B7HF6FJKEBOZNZK2EO4I3PIU65AQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.552138956 (625h25m4.423221992s) [noise=true , address=145.79.227.38:4977 , piece_id=LZHY2WNZ6X23TN65R4PVJWLDYV4K57BQ4C6LRPXHVYNYIEO57S7Q, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.552418960 (625h25m4.423502005s) [noise=true , address=145.79.227.4:4185 , piece_id=RI6CLWZ2ZXPQ36SWRQTRI3ECAFTSHUK5TWFKYYXTE7CERSD3YGIQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.552838599 (625h25m4.423921644s) [noise=true , address=45.91.116.8:4046 , piece_id=YKXTIPM2QQD45UADUYGAUXH4TQFJII3SUKJ3W5NMYJYLNF4O57TA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.554332483 (625h25m4.425415528s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.554476558 (625h25m4.425559593s) [noise=true , address=77.90.31.10:4517 , piece_id=NDSDOK2KXTFHPZ237OIQOEKEWR5PAEJKN6L4RFXSYNHFXISGYHPA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.555638550 (625h25m4.426721585s) [noise=true , address=145.79.227.9:4367 , piece_id=MGWDE3UGN7OTFEHKEZKF3CTD4WMZWOXYIKU7543ZXQAXHHBQTI7Q, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.556990394 (625h25m4.428073439s) [noise=true , address=77.90.31.15:4697 , piece_id=MNW477DNTP25EHJMHVVW3BX3UYDBJGG7FXHFSTFZLTM2ZW4NV4ZQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.565033940 (625h25m4.436116975s) [noise=true , address=45.91.116.14:4191 , piece_id=HSXBPP3II2CHJHX3BFXR4BNCTAQDGLIRGBAPGHV7ZUP5IDMBTDQA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.565184408 (625h25m4.436267443s) [noise=true , address=145.79.227.14:4837 , piece_id=JHFAKK2Q6GUN3LSB7OLZE4QH5BQ3V2ANPT4QG6FQ7XC3SYPXSFLQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.566499170 (625h25m4.437582215s) [noise=true , address=145.79.227.52:4445 , piece_id=KLVNKKAVLTCDXBPGQDFLEOXOODU2KS4MPO4Y6OEPSVMX4AXRRWEQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.567394656 (625h25m4.438477701s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:06.264153690 (625h25m6.135236735s) [upload=548032]: Next returned nil segment
Dec 31 16:54:06.264177514 (625h25m6.135260559s) [upload=548032]: Waiting for error group managing segments...
Dec 31 16:54:06.536331012 (625h25m6.407414047s) [noise=true , address=145.79.227.52:4445 , piece_id=KLVNKKAVLTCDXBPGQDFLEOXOODU2KS4MPO4Y6OEPSVMX4AXRRWEQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.538792870 (625h25m6.409875906s) [noise=true , address=145.79.227.15:4874 , piece_id=3CVWADJWSRXY6B5SR5NCXK2ROZKQDUWX66NPHV3B3HN3AC72666Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.559727663 (625h25m6.430810698s) [noise=true , address=145.79.227.4:4185 , piece_id=RI6CLWZ2ZXPQ36SWRQTRI3ECAFTSHUK5TWFKYYXTE7CERSD3YGIQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.560250448 (625h25m6.431333493s) [noise=true , address=145.79.227.9:4367 , piece_id=MGWDE3UGN7OTFEHKEZKF3CTD4WMZWOXYIKU7543ZXQAXHHBQTI7Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.569094128 (625h25m6.440177173s) [noise=true , address=145.79.227.14:4837 , piece_id=JHFAKK2Q6GUN3LSB7OLZE4QH5BQ3V2ANPT4QG6FQ7XC3SYPXSFLQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.626771975 (625h25m6.49785501s) [noise=true , address=145.79.227.38:4977 , piece_id=LZHY2WNZ6X23TN65R4PVJWLDYV4K57BQ4C6LRPXHVYNYIEO57S7Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.636667477 (625h25m6.507750522s) [noise=true , address=145.79.227.19:4079 , piece_id=TYQFG66ZLQRS5OTMLNOYRR24B7HF6FJKEBOZNZK2EO4I3PIU65AQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.741721214 (625h25m6.612804259s) [noise=true , address=77.90.31.10:4517 , piece_id=NDSDOK2KXTFHPZ237OIQOEKEWR5PAEJKN6L4RFXSYNHFXISGYHPA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.863029281 (625h25m6.734112316s) [noise=true , address=45.91.116.13:4153 , piece_id=MNIXKBZDEUHYK2Q66CVJFJJDNJMRICPWD4QVUZNKBOVSMKXIW4EA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.904267314 (625h25m6.775350349s) [noise=true , address=77.90.31.15:4697 , piece_id=MNW477DNTP25EHJMHVVW3BX3UYDBJGG7FXHFSTFZLTM2ZW4NV4ZQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.999818748 (625h25m6.870901783s) [noise=true , address=45.91.116.14:4191 , piece_id=HSXBPP3II2CHJHX3BFXR4BNCTAQDGLIRGBAPGHV7ZUP5IDMBTDQA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:07.503007990 (625h25m7.374091025s) [noise=true , address=77.90.31.14:4662 , piece_id=Y3JXTJ3RYGB2JNDSRGS4XDA5BOVOS6WEGK44HZLYYLDTPOQXXQTA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:07.956546069 (625h25m7.827629114s) [noise=true , address=45.91.116.8:4046 , piece_id=YKXTIPM2QQD45UADUYGAUXH4TQFJII3SUKJ3W5NMYJYLNF4O57TA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:07.956571196 (625h25m7.827654231s) [seg_pos={0 0}, upload=548032]: Segment reached optimal threshold of 13 pieces.
Dec 31 16:54:07.956812125 (625h25m7.82789517s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF
Dec 31 16:54:07.956823177 (625h25m7.827906222s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold
Dec 31 16:54:07.957133406 (625h25m7.828216442s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF
Dec 31 16:54:07.957148957 (625h25m7.828231992s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold
Dec 31 16:54:07.957231634 (625h25m7.828314689s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF
Dec 31 16:54:07.957242886 (625h25m7.828325931s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold
Dec 31 16:54:07.957268374 (625h25m7.828351409s) [seg_pos={0 0}, upload=548032]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 16:54:07.957349389 (625h25m7.828432434s) [upload=548032]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:54:07.957356853 (625h25m7.828439898s) [upload=548032]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:54:08.248058148 (625h25m8.119141213s) [upload=548032]: Done uploading segments...
Dec 31 16:54:08.248084378 (625h25m8.119167413s) [upload=548032]: Upload finished. err: <nil>
Duration: 6.073111202s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....X..........X.....................................X.....] duration:
38.178850496s trace:
https://jaeger.dev.storj.io/trace/1fe9f44dea5d609c
FAIL (2025-12-31 16:55:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
.241246563 (625h22m16.112329598s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.241981543 (625h22m16.113064588s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242111701 (625h22m16.113194746s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242433975 (625h22m16.11351702s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242713457 (625h22m16.113796492s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242802286 (625h22m16.113885331s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.243245350 (625h22m16.114328395s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.243631865 (625h22m16.114714901s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.244008643 (625h22m16.115091678s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.244381102 (625h22m16.115464147s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.244758581 (625h22m16.115841616s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.245176256 (625h22m16.116259301s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.245538285 (625h22m16.11662133s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.245960649 (625h22m16.117043684s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.246337136 (625h22m16.117420181s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.247466537 (625h22m16.118549572s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.247996927 (625h22m16.119079962s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:17.693671241 (625h22m17.564754286s) [upload=547992]: Next returned nil segment
Dec 31 16:51:17.693714864 (625h22m17.564797899s) [upload=547992]: Waiting for error group managing segments...
Dec 31 16:51:18.120111632 (625h22m17.991194667s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.122023212 (625h22m17.993106257s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.201357746 (625h22m18.072440781s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.293128900 (625h22m18.164211965s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.306933417 (625h22m18.178016452s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.357131932 (625h22m18.228214977s) [noise=true , address=77.90.31.11:4563 , piece_id=2LRC66FCB4DTXU44NFN5O5L6ZMGDCQ6LHHB7TLYJ2LZSYKRWJEYQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.390427070 (625h22m18.261510105s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.469358907 (625h22m18.340441942s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.675586202 (625h22m18.546669267s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.834344079 (625h22m18.705427114s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.845713768 (625h22m18.716796813s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.980140516 (625h22m18.851223551s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.980522623 (625h22m18.851605658s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.997884583 (625h22m18.868967618s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.997904631 (625h22m18.868987666s) [seg_pos={0 0}, upload=547992]: Segment reached optimal threshold of 14 pieces.
Dec 31 16:51:18.998092839 (625h22m18.869175874s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998099372 (625h22m18.869182417s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998204202 (625h22m18.869287237s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998210183 (625h22m18.869293218s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998213039 (625h22m18.869296074s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998221765 (625h22m18.86930481s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998312208 (625h22m18.869395243s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122S3PRcJwYfK5UEWESnYA7aTPHA6zCLFyBuFSm2Pb5wFSxDPiR): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998318149 (625h22m18.869401184s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998356041 (625h22m18.869439076s) [seg_pos={0 0}, upload=547992]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 16:51:18.998428418 (625h22m18.869511454s) [upload=547992]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:51:18.998433228 (625h22m18.869516273s) [upload=547992]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:51:19.285646870 (625h22m19.156729935s) [upload=547992]: Done uploading segments...
Dec 31 16:51:19.285688400 (625h22m19.156771445s) [upload=547992]: Upload finished. err: <nil>
Duration: 38.178873461s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
34 out of 60 is failed (threshold=6) [X....XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....X.X] duration:
5.002121078s trace:
https://jaeger.dev.storj.io/trace/5e25e590cc565f32
FAIL (2025-12-31 16:55:33 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002146186s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
6.916457127s trace:
https://jaeger.dev.storj.io/trace/5582e72abe7a8819
FAIL (2025-12-31 16:55:10 UTC)
orbiter warning: context deadline exceeded
Duration: 6.916477025s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [.X....XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....X.] duration:
3.538687078s trace:
https://jaeger.dev.storj.io/trace/10c21ab25da0f02f
FAIL (2025-12-31 16:55:04 UTC)
orbiter warning: context deadline exceeded
Duration: 3.538721192s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [......X..........X.....................................X....] duration:
30.58543632s trace:
https://jaeger.dev.storj.io/trace/1fe9f44dea5d609c
FAIL (2025-12-31 16:55:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
.241246563 (625h22m16.112329598s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.241981543 (625h22m16.113064588s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242111701 (625h22m16.113194746s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242433975 (625h22m16.11351702s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242713457 (625h22m16.113796492s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242802286 (625h22m16.113885331s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.243245350 (625h22m16.114328395s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.243631865 (625h22m16.114714901s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.244008643 (625h22m16.115091678s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.244381102 (625h22m16.115464147s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.244758581 (625h22m16.115841616s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.245176256 (625h22m16.116259301s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.245538285 (625h22m16.11662133s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.245960649 (625h22m16.117043684s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.246337136 (625h22m16.117420181s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.247466537 (625h22m16.118549572s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.247996927 (625h22m16.119079962s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:17.693671241 (625h22m17.564754286s) [upload=547992]: Next returned nil segment
Dec 31 16:51:17.693714864 (625h22m17.564797899s) [upload=547992]: Waiting for error group managing segments...
Dec 31 16:51:18.120111632 (625h22m17.991194667s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.122023212 (625h22m17.993106257s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.201357746 (625h22m18.072440781s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.293128900 (625h22m18.164211965s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.306933417 (625h22m18.178016452s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.357131932 (625h22m18.228214977s) [noise=true , address=77.90.31.11:4563 , piece_id=2LRC66FCB4DTXU44NFN5O5L6ZMGDCQ6LHHB7TLYJ2LZSYKRWJEYQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.390427070 (625h22m18.261510105s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.469358907 (625h22m18.340441942s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.675586202 (625h22m18.546669267s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.834344079 (625h22m18.705427114s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.845713768 (625h22m18.716796813s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.980140516 (625h22m18.851223551s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.980522623 (625h22m18.851605658s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.997884583 (625h22m18.868967618s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.997904631 (625h22m18.868987666s) [seg_pos={0 0}, upload=547992]: Segment reached optimal threshold of 14 pieces.
Dec 31 16:51:18.998092839 (625h22m18.869175874s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998099372 (625h22m18.869182417s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998204202 (625h22m18.869287237s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998210183 (625h22m18.869293218s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998213039 (625h22m18.869296074s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998221765 (625h22m18.86930481s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998312208 (625h22m18.869395243s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122S3PRcJwYfK5UEWESnYA7aTPHA6zCLFyBuFSm2Pb5wFSxDPiR): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998318149 (625h22m18.869401184s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998356041 (625h22m18.869439076s) [seg_pos={0 0}, upload=547992]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 16:51:18.998428418 (625h22m18.869511454s) [upload=547992]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:51:18.998433228 (625h22m18.869516273s) [upload=547992]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:51:19.285646870 (625h22m19.156729935s) [upload=547992]: Done uploading segments...
Dec 31 16:51:19.285688400 (625h22m19.156771445s) [upload=547992]: Upload finished. err: <nil>
Duration: 30.585470122s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [..XX........X....X.X...X....X.X.......X.XX..X..X...X.X.XX.X.] duration:
6.052573762s trace:
https://jaeger.dev.storj.io/trace/2357ddb7e2c2e281
FAIL (2025-12-31 16:55:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 16:54:04.303380506 (625h25m4.174463542s) [upload=548032]: Starting upload
Dec 31 16:54:04.303432836 (625h25m4.174515871s) [upload=548032]: Done starting upload
Dec 31 16:54:04.303545921 (625h25m4.174628976s) [upload=548032]: Uploading segments...
Dec 31 16:54:04.303631374 (625h25m4.174714449s) [upload=548032]: Got next segment. Inline: false
Dec 31 16:54:04.303679495 (625h25m4.17476254s) [upload=548032]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 16:54:04.303700816 (625h25m4.174783851s) [upload=548032]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 16:54:04.550160820 (625h25m4.421243865s) [seg_pos={0 0}, upload=548032]: Begin upload segment...
Dec 31 16:54:04.550206296 (625h25m4.421289341s) [seg_pos={0 0}, upload=548032]: Done begin upload segment.
Dec 31 16:54:04.550233237 (625h25m4.421316272s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.550319773 (625h25m4.421402848s) [noise=true , address=145.79.227.15:4874 , piece_id=3CVWADJWSRXY6B5SR5NCXK2ROZKQDUWX66NPHV3B3HN3AC72666Q, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.550895328 (625h25m4.421978363s) [noise=true , address=77.90.31.14:4662 , piece_id=Y3JXTJ3RYGB2JNDSRGS4XDA5BOVOS6WEGK44HZLYYLDTPOQXXQTA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.551320608 (625h25m4.422403653s) [noise=true , address=45.91.116.13:4153 , piece_id=MNIXKBZDEUHYK2Q66CVJFJJDNJMRICPWD4QVUZNKBOVSMKXIW4EA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.551783460 (625h25m4.422866495s) [noise=true , address=145.79.227.19:4079 , piece_id=TYQFG66ZLQRS5OTMLNOYRR24B7HF6FJKEBOZNZK2EO4I3PIU65AQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.552138956 (625h25m4.423221992s) [noise=true , address=145.79.227.38:4977 , piece_id=LZHY2WNZ6X23TN65R4PVJWLDYV4K57BQ4C6LRPXHVYNYIEO57S7Q, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.552418960 (625h25m4.423502005s) [noise=true , address=145.79.227.4:4185 , piece_id=RI6CLWZ2ZXPQ36SWRQTRI3ECAFTSHUK5TWFKYYXTE7CERSD3YGIQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.552838599 (625h25m4.423921644s) [noise=true , address=45.91.116.8:4046 , piece_id=YKXTIPM2QQD45UADUYGAUXH4TQFJII3SUKJ3W5NMYJYLNF4O57TA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.554332483 (625h25m4.425415528s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.554476558 (625h25m4.425559593s) [noise=true , address=77.90.31.10:4517 , piece_id=NDSDOK2KXTFHPZ237OIQOEKEWR5PAEJKN6L4RFXSYNHFXISGYHPA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.555638550 (625h25m4.426721585s) [noise=true , address=145.79.227.9:4367 , piece_id=MGWDE3UGN7OTFEHKEZKF3CTD4WMZWOXYIKU7543ZXQAXHHBQTI7Q, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.556990394 (625h25m4.428073439s) [noise=true , address=77.90.31.15:4697 , piece_id=MNW477DNTP25EHJMHVVW3BX3UYDBJGG7FXHFSTFZLTM2ZW4NV4ZQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.565033940 (625h25m4.436116975s) [noise=true , address=45.91.116.14:4191 , piece_id=HSXBPP3II2CHJHX3BFXR4BNCTAQDGLIRGBAPGHV7ZUP5IDMBTDQA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.565184408 (625h25m4.436267443s) [noise=true , address=145.79.227.14:4837 , piece_id=JHFAKK2Q6GUN3LSB7OLZE4QH5BQ3V2ANPT4QG6FQ7XC3SYPXSFLQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.566499170 (625h25m4.437582215s) [noise=true , address=145.79.227.52:4445 , piece_id=KLVNKKAVLTCDXBPGQDFLEOXOODU2KS4MPO4Y6OEPSVMX4AXRRWEQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.567394656 (625h25m4.438477701s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:06.264153690 (625h25m6.135236735s) [upload=548032]: Next returned nil segment
Dec 31 16:54:06.264177514 (625h25m6.135260559s) [upload=548032]: Waiting for error group managing segments...
Dec 31 16:54:06.536331012 (625h25m6.407414047s) [noise=true , address=145.79.227.52:4445 , piece_id=KLVNKKAVLTCDXBPGQDFLEOXOODU2KS4MPO4Y6OEPSVMX4AXRRWEQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.538792870 (625h25m6.409875906s) [noise=true , address=145.79.227.15:4874 , piece_id=3CVWADJWSRXY6B5SR5NCXK2ROZKQDUWX66NPHV3B3HN3AC72666Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.559727663 (625h25m6.430810698s) [noise=true , address=145.79.227.4:4185 , piece_id=RI6CLWZ2ZXPQ36SWRQTRI3ECAFTSHUK5TWFKYYXTE7CERSD3YGIQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.560250448 (625h25m6.431333493s) [noise=true , address=145.79.227.9:4367 , piece_id=MGWDE3UGN7OTFEHKEZKF3CTD4WMZWOXYIKU7543ZXQAXHHBQTI7Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.569094128 (625h25m6.440177173s) [noise=true , address=145.79.227.14:4837 , piece_id=JHFAKK2Q6GUN3LSB7OLZE4QH5BQ3V2ANPT4QG6FQ7XC3SYPXSFLQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.626771975 (625h25m6.49785501s) [noise=true , address=145.79.227.38:4977 , piece_id=LZHY2WNZ6X23TN65R4PVJWLDYV4K57BQ4C6LRPXHVYNYIEO57S7Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.636667477 (625h25m6.507750522s) [noise=true , address=145.79.227.19:4079 , piece_id=TYQFG66ZLQRS5OTMLNOYRR24B7HF6FJKEBOZNZK2EO4I3PIU65AQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.741721214 (625h25m6.612804259s) [noise=true , address=77.90.31.10:4517 , piece_id=NDSDOK2KXTFHPZ237OIQOEKEWR5PAEJKN6L4RFXSYNHFXISGYHPA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.863029281 (625h25m6.734112316s) [noise=true , address=45.91.116.13:4153 , piece_id=MNIXKBZDEUHYK2Q66CVJFJJDNJMRICPWD4QVUZNKBOVSMKXIW4EA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.904267314 (625h25m6.775350349s) [noise=true , address=77.90.31.15:4697 , piece_id=MNW477DNTP25EHJMHVVW3BX3UYDBJGG7FXHFSTFZLTM2ZW4NV4ZQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.999818748 (625h25m6.870901783s) [noise=true , address=45.91.116.14:4191 , piece_id=HSXBPP3II2CHJHX3BFXR4BNCTAQDGLIRGBAPGHV7ZUP5IDMBTDQA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:07.503007990 (625h25m7.374091025s) [noise=true , address=77.90.31.14:4662 , piece_id=Y3JXTJ3RYGB2JNDSRGS4XDA5BOVOS6WEGK44HZLYYLDTPOQXXQTA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:07.956546069 (625h25m7.827629114s) [noise=true , address=45.91.116.8:4046 , piece_id=YKXTIPM2QQD45UADUYGAUXH4TQFJII3SUKJ3W5NMYJYLNF4O57TA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:07.956571196 (625h25m7.827654231s) [seg_pos={0 0}, upload=548032]: Segment reached optimal threshold of 13 pieces.
Dec 31 16:54:07.956812125 (625h25m7.82789517s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF
Dec 31 16:54:07.956823177 (625h25m7.827906222s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold
Dec 31 16:54:07.957133406 (625h25m7.828216442s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF
Dec 31 16:54:07.957148957 (625h25m7.828231992s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold
Dec 31 16:54:07.957231634 (625h25m7.828314689s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF
Dec 31 16:54:07.957242886 (625h25m7.828325931s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold
Dec 31 16:54:07.957268374 (625h25m7.828351409s) [seg_pos={0 0}, upload=548032]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 16:54:07.957349389 (625h25m7.828432434s) [upload=548032]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:54:07.957356853 (625h25m7.828439898s) [upload=548032]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:54:08.248058148 (625h25m8.119141213s) [upload=548032]: Done uploading segments...
Dec 31 16:54:08.248084378 (625h25m8.119167413s) [upload=548032]: Upload finished. err: <nil>
Duration: 6.052634036s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [..X....XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....X] duration:
5.00092986s trace:
https://jaeger.dev.storj.io/trace/10c21ab25da0f02f
FAIL (2025-12-31 16:54:42 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000951802s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [X..XX........X....X.X...X....X.X.......X.XX..X..X...X.X.XX.X] duration:
30.002624421s trace:
https://jaeger.dev.storj.io/trace/2357ddb7e2c2e281
FAIL (2025-12-31 16:54:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 16:54:04.303380506 (625h25m4.174463542s) [upload=548032]: Starting upload
Dec 31 16:54:04.303432836 (625h25m4.174515871s) [upload=548032]: Done starting upload
Dec 31 16:54:04.303545921 (625h25m4.174628976s) [upload=548032]: Uploading segments...
Dec 31 16:54:04.303631374 (625h25m4.174714449s) [upload=548032]: Got next segment. Inline: false
Dec 31 16:54:04.303679495 (625h25m4.17476254s) [upload=548032]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 16:54:04.303700816 (625h25m4.174783851s) [upload=548032]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 16:54:04.550160820 (625h25m4.421243865s) [seg_pos={0 0}, upload=548032]: Begin upload segment...
Dec 31 16:54:04.550206296 (625h25m4.421289341s) [seg_pos={0 0}, upload=548032]: Done begin upload segment.
Dec 31 16:54:04.550233237 (625h25m4.421316272s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.550319773 (625h25m4.421402848s) [noise=true , address=145.79.227.15:4874 , piece_id=3CVWADJWSRXY6B5SR5NCXK2ROZKQDUWX66NPHV3B3HN3AC72666Q, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.550895328 (625h25m4.421978363s) [noise=true , address=77.90.31.14:4662 , piece_id=Y3JXTJ3RYGB2JNDSRGS4XDA5BOVOS6WEGK44HZLYYLDTPOQXXQTA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.551320608 (625h25m4.422403653s) [noise=true , address=45.91.116.13:4153 , piece_id=MNIXKBZDEUHYK2Q66CVJFJJDNJMRICPWD4QVUZNKBOVSMKXIW4EA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.551783460 (625h25m4.422866495s) [noise=true , address=145.79.227.19:4079 , piece_id=TYQFG66ZLQRS5OTMLNOYRR24B7HF6FJKEBOZNZK2EO4I3PIU65AQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.552138956 (625h25m4.423221992s) [noise=true , address=145.79.227.38:4977 , piece_id=LZHY2WNZ6X23TN65R4PVJWLDYV4K57BQ4C6LRPXHVYNYIEO57S7Q, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.552418960 (625h25m4.423502005s) [noise=true , address=145.79.227.4:4185 , piece_id=RI6CLWZ2ZXPQ36SWRQTRI3ECAFTSHUK5TWFKYYXTE7CERSD3YGIQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.552838599 (625h25m4.423921644s) [noise=true , address=45.91.116.8:4046 , piece_id=YKXTIPM2QQD45UADUYGAUXH4TQFJII3SUKJ3W5NMYJYLNF4O57TA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.554332483 (625h25m4.425415528s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.554476558 (625h25m4.425559593s) [noise=true , address=77.90.31.10:4517 , piece_id=NDSDOK2KXTFHPZ237OIQOEKEWR5PAEJKN6L4RFXSYNHFXISGYHPA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.555638550 (625h25m4.426721585s) [noise=true , address=145.79.227.9:4367 , piece_id=MGWDE3UGN7OTFEHKEZKF3CTD4WMZWOXYIKU7543ZXQAXHHBQTI7Q, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.556990394 (625h25m4.428073439s) [noise=true , address=77.90.31.15:4697 , piece_id=MNW477DNTP25EHJMHVVW3BX3UYDBJGG7FXHFSTFZLTM2ZW4NV4ZQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.565033940 (625h25m4.436116975s) [noise=true , address=45.91.116.14:4191 , piece_id=HSXBPP3II2CHJHX3BFXR4BNCTAQDGLIRGBAPGHV7ZUP5IDMBTDQA, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.565184408 (625h25m4.436267443s) [noise=true , address=145.79.227.14:4837 , piece_id=JHFAKK2Q6GUN3LSB7OLZE4QH5BQ3V2ANPT4QG6FQ7XC3SYPXSFLQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.566499170 (625h25m4.437582215s) [noise=true , address=145.79.227.52:4445 , piece_id=KLVNKKAVLTCDXBPGQDFLEOXOODU2KS4MPO4Y6OEPSVMX4AXRRWEQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:04.567394656 (625h25m4.438477701s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Uploading piece...
Dec 31 16:54:06.264153690 (625h25m6.135236735s) [upload=548032]: Next returned nil segment
Dec 31 16:54:06.264177514 (625h25m6.135260559s) [upload=548032]: Waiting for error group managing segments...
Dec 31 16:54:06.536331012 (625h25m6.407414047s) [noise=true , address=145.79.227.52:4445 , piece_id=KLVNKKAVLTCDXBPGQDFLEOXOODU2KS4MPO4Y6OEPSVMX4AXRRWEQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.538792870 (625h25m6.409875906s) [noise=true , address=145.79.227.15:4874 , piece_id=3CVWADJWSRXY6B5SR5NCXK2ROZKQDUWX66NPHV3B3HN3AC72666Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.559727663 (625h25m6.430810698s) [noise=true , address=145.79.227.4:4185 , piece_id=RI6CLWZ2ZXPQ36SWRQTRI3ECAFTSHUK5TWFKYYXTE7CERSD3YGIQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.560250448 (625h25m6.431333493s) [noise=true , address=145.79.227.9:4367 , piece_id=MGWDE3UGN7OTFEHKEZKF3CTD4WMZWOXYIKU7543ZXQAXHHBQTI7Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.569094128 (625h25m6.440177173s) [noise=true , address=145.79.227.14:4837 , piece_id=JHFAKK2Q6GUN3LSB7OLZE4QH5BQ3V2ANPT4QG6FQ7XC3SYPXSFLQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.626771975 (625h25m6.49785501s) [noise=true , address=145.79.227.38:4977 , piece_id=LZHY2WNZ6X23TN65R4PVJWLDYV4K57BQ4C6LRPXHVYNYIEO57S7Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.636667477 (625h25m6.507750522s) [noise=true , address=145.79.227.19:4079 , piece_id=TYQFG66ZLQRS5OTMLNOYRR24B7HF6FJKEBOZNZK2EO4I3PIU65AQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.741721214 (625h25m6.612804259s) [noise=true , address=77.90.31.10:4517 , piece_id=NDSDOK2KXTFHPZ237OIQOEKEWR5PAEJKN6L4RFXSYNHFXISGYHPA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.863029281 (625h25m6.734112316s) [noise=true , address=45.91.116.13:4153 , piece_id=MNIXKBZDEUHYK2Q66CVJFJJDNJMRICPWD4QVUZNKBOVSMKXIW4EA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.904267314 (625h25m6.775350349s) [noise=true , address=77.90.31.15:4697 , piece_id=MNW477DNTP25EHJMHVVW3BX3UYDBJGG7FXHFSTFZLTM2ZW4NV4ZQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:06.999818748 (625h25m6.870901783s) [noise=true , address=45.91.116.14:4191 , piece_id=HSXBPP3II2CHJHX3BFXR4BNCTAQDGLIRGBAPGHV7ZUP5IDMBTDQA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:07.503007990 (625h25m7.374091025s) [noise=true , address=77.90.31.14:4662 , piece_id=Y3JXTJ3RYGB2JNDSRGS4XDA5BOVOS6WEGK44HZLYYLDTPOQXXQTA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:07.956546069 (625h25m7.827629114s) [noise=true , address=45.91.116.8:4046 , piece_id=YKXTIPM2QQD45UADUYGAUXH4TQFJII3SUKJ3W5NMYJYLNF4O57TA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil>
Dec 31 16:54:07.956571196 (625h25m7.827654231s) [seg_pos={0 0}, upload=548032]: Segment reached optimal threshold of 13 pieces.
Dec 31 16:54:07.956812125 (625h25m7.82789517s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF
Dec 31 16:54:07.956823177 (625h25m7.827906222s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold
Dec 31 16:54:07.957133406 (625h25m7.828216442s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF
Dec 31 16:54:07.957148957 (625h25m7.828231992s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold
Dec 31 16:54:07.957231634 (625h25m7.828314689s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF
Dec 31 16:54:07.957242886 (625h25m7.828325931s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold
Dec 31 16:54:07.957268374 (625h25m7.828351409s) [seg_pos={0 0}, upload=548032]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 16:54:07.957349389 (625h25m7.828432434s) [upload=548032]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:54:07.957356853 (625h25m7.828439898s) [upload=548032]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:54:08.248058148 (625h25m8.119141213s) [upload=548032]: Done uploading segments...
Dec 31 16:54:08.248084378 (625h25m8.119167413s) [upload=548032]: Upload finished. err: <nil>
Duration: 30.002645482s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.938244412s trace:
https://jaeger.dev.storj.io/trace/5582e72abe7a8819
FAIL (2025-12-31 16:54:18 UTC)
orbiter warning: context deadline exceeded
Duration: 5.938266224s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......X..........X.....................................X...] duration:
44.945188361s trace:
https://jaeger.dev.storj.io/trace/1fe9f44dea5d609c
FAIL (2025-12-31 16:54:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
.241246563 (625h22m16.112329598s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.241981543 (625h22m16.113064588s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242111701 (625h22m16.113194746s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242433975 (625h22m16.11351702s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242713457 (625h22m16.113796492s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242802286 (625h22m16.113885331s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.243245350 (625h22m16.114328395s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.243631865 (625h22m16.114714901s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.244008643 (625h22m16.115091678s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.244381102 (625h22m16.115464147s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.244758581 (625h22m16.115841616s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.245176256 (625h22m16.116259301s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.245538285 (625h22m16.11662133s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.245960649 (625h22m16.117043684s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.246337136 (625h22m16.117420181s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.247466537 (625h22m16.118549572s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.247996927 (625h22m16.119079962s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:17.693671241 (625h22m17.564754286s) [upload=547992]: Next returned nil segment
Dec 31 16:51:17.693714864 (625h22m17.564797899s) [upload=547992]: Waiting for error group managing segments...
Dec 31 16:51:18.120111632 (625h22m17.991194667s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.122023212 (625h22m17.993106257s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.201357746 (625h22m18.072440781s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.293128900 (625h22m18.164211965s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.306933417 (625h22m18.178016452s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.357131932 (625h22m18.228214977s) [noise=true , address=77.90.31.11:4563 , piece_id=2LRC66FCB4DTXU44NFN5O5L6ZMGDCQ6LHHB7TLYJ2LZSYKRWJEYQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.390427070 (625h22m18.261510105s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.469358907 (625h22m18.340441942s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.675586202 (625h22m18.546669267s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.834344079 (625h22m18.705427114s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.845713768 (625h22m18.716796813s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.980140516 (625h22m18.851223551s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.980522623 (625h22m18.851605658s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.997884583 (625h22m18.868967618s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.997904631 (625h22m18.868987666s) [seg_pos={0 0}, upload=547992]: Segment reached optimal threshold of 14 pieces.
Dec 31 16:51:18.998092839 (625h22m18.869175874s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998099372 (625h22m18.869182417s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998204202 (625h22m18.869287237s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998210183 (625h22m18.869293218s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998213039 (625h22m18.869296074s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998221765 (625h22m18.86930481s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998312208 (625h22m18.869395243s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122S3PRcJwYfK5UEWESnYA7aTPHA6zCLFyBuFSm2Pb5wFSxDPiR): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998318149 (625h22m18.869401184s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998356041 (625h22m18.869439076s) [seg_pos={0 0}, upload=547992]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 16:51:18.998428418 (625h22m18.869511454s) [upload=547992]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:51:18.998433228 (625h22m18.869516273s) [upload=547992]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:51:19.285646870 (625h22m19.156729935s) [upload=547992]: Done uploading segments...
Dec 31 16:51:19.285688400 (625h22m19.156771445s) [upload=547992]: Upload finished. err: <nil>
Duration: 44.945203308s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
32 out of 60 is failed (threshold=6) [...X....XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....] duration:
3.533593772s trace:
https://jaeger.dev.storj.io/trace/19c6aa42963b3c0f
FAIL (2025-12-31 16:53:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.53362424s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [XX..XX........X....X.X...X....X.X.......X.XX..X..X...X.X.XX.] duration:
8.095222294s trace:
https://jaeger.dev.storj.io/trace/5cfa15b3395c99c7
FAIL (2025-12-31 16:53:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: HVYJJTDKP2PNFD4EPFN3NV5O3EPMD5REU3VF5KPAOZRMSSL3REQQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: NZYM32SUJY5H5FVGJQL3ZBQZNGVJVBQMU7O4SRZX6RXPBK34NRFA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 72ZVQSHOUQHKQUBCGG3UZDQWLLDSTCHQR7RRZYARVWSMVVGWPEYA) context deadline exceeded
======LOGS======
(625h24m0.187935032s) [upload=548018]: Uploading segments...
Dec 31 16:53:00.317161005 (625h24m0.18824405s) [upload=548018]: Got next segment. Inline: false
Dec 31 16:53:00.317208526 (625h24m0.188291572s) [upload=548018]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 16:53:00.317214959 (625h24m0.188298004s) [upload=548018]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 16:53:00.559124607 (625h24m0.430207652s) [seg_pos={0 0}, upload=548018]: Begin upload segment...
Dec 31 16:53:00.559295773 (625h24m0.430378808s) [seg_pos={0 0}, upload=548018]: Done begin upload segment.
Dec 31 16:53:00.559331241 (625h24m0.430414276s) [noise=true , address=77.90.31.14:4663 , piece_id=HVYJJTDKP2PNFD4EPFN3NV5O3EPMD5REU3VF5KPAOZRMSSL3REQQ, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.560814085 (625h24m0.43189712s) [noise=true , address=145.79.227.22:4511 , piece_id=COUKMWSBDOUPUV6IBMTMM7MDRLJFR2QVHI4P6TD6K7XXLDKTID5Q, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.561408707 (625h24m0.432491752s) [noise=true , address=145.79.227.8:4330 , piece_id=XG67YYMPMX3ICYAE7AGFG4EFFIQILXFPEL563MCXVXCFX243LKBQ, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.561720671 (625h24m0.432803716s) [noise=true , address=145.79.227.41:4087 , piece_id=2NMUN4BSMAGHN4VFZM3AEQWHZN6S2IGMKSURL5EKEPB4BIIUAZFA, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.561960947 (625h24m0.433043982s) [noise=true , address=145.79.227.52:4446 , piece_id=RQCOSCULWHAOXN662DD7G5OYXFWYWF7K6ZASKM3DFJO35WTGBWAA, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.562345741 (625h24m0.433428786s) [noise=true , address=45.91.116.7:4012 , piece_id=NZYM32SUJY5H5FVGJQL3ZBQZNGVJVBQMU7O4SRZX6RXPBK34NRFA, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.562382371 (625h24m0.433465416s) [noise=true , address=45.91.116.11:4118 , piece_id=2J6BXGMDT4EHSNS5IPGT4RUOFKFLJ56OFV4HUJUJA2SBYJ3JBK2Q, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.562671952 (625h24m0.433754997s) [noise=true , address=45.91.116.13:4154 , piece_id=HUUYW6JZ6RQHLX6K5QOPNOLA4WHR2QJZBLF6XAU2KG2GHIIXU67A, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.562940604 (625h24m0.434023649s) [noise=true , address=145.79.227.39:4015 , piece_id=E3NXLXBM3IIK7ICGMIOGAZOGUVE24U4O3S5TLV5OFJAGD6NRGJVA, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.562982724 (625h24m0.434065799s) [noise=true , address=45.91.116.10:4082 , piece_id=72ZVQSHOUQHKQUBCGG3UZDQWLLDSTCHQR7RRZYARVWSMVVGWPEYA, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.563555213 (625h24m0.434638258s) [noise=true , address=145.79.227.38:4978 , piece_id=BOVSXWQR7323NSG3D2T2RNWG25TYTRQQ7C4RCP7RVEOM65HXFPLQ, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.564216943 (625h24m0.435299988s) [noise=true , address=145.79.227.42:4123 , piece_id=2KOMXT4FVOLDWYLFFTOT3Z7QRC2SQPUMY3WJROPBAS6EV7FNSTIA, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.564915253 (625h24m0.435998288s) [noise=true , address=45.91.116.9:4261 , piece_id=DKGELPLYLSA46I6OHE4K3NNLIJLRB66DNFCQK7HVYIPP54L5RZGQ, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.565695830 (625h24m0.436778875s) [noise=true , address=145.79.227.32:4835 , piece_id=ZUMYIO55NMELJXSHOJSDXGEMGFEXYTAOFTQUCAWL2CW4UFZFRHMQ, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.565999679 (625h24m0.437082714s) [noise=true , address=145.79.227.3:4150 , piece_id=IICUES4EAAPXXEUZHMS5M742ATNEZ6ZH2L6343NBEFWKN5A4EDTA, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.566426211 (625h24m0.437509246s) [noise=true , address=145.79.227.20:4437 , piece_id=ZB5POSRWAFSXFQBHGP6BNWH7FV6SHBUNWPB5M2GBGHFEZ72P5P6A, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:02.269582063 (625h24m2.140665109s) [upload=548018]: Next returned nil segment
Dec 31 16:53:02.269601401 (625h24m2.140684456s) [upload=548018]: Waiting for error group managing segments...
Dec 31 16:53:02.550702465 (625h24m2.42178551s) [noise=true , address=145.79.227.20:4437 , piece_id=ZB5POSRWAFSXFQBHGP6BNWH7FV6SHBUNWPB5M2GBGHFEZ72P5P6A, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.554501710 (625h24m2.425584765s) [noise=true , address=145.79.227.52:4446 , piece_id=RQCOSCULWHAOXN662DD7G5OYXFWYWF7K6ZASKM3DFJO35WTGBWAA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.569282905 (625h24m2.44036595s) [noise=true , address=145.79.227.3:4150 , piece_id=IICUES4EAAPXXEUZHMS5M742ATNEZ6ZH2L6343NBEFWKN5A4EDTA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.572389480 (625h24m2.443472526s) [noise=true , address=145.79.227.22:4511 , piece_id=COUKMWSBDOUPUV6IBMTMM7MDRLJFR2QVHI4P6TD6K7XXLDKTID5Q, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.579231618 (625h24m2.450314664s) [noise=true , address=145.79.227.8:4330 , piece_id=XG67YYMPMX3ICYAE7AGFG4EFFIQILXFPEL563MCXVXCFX243LKBQ, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.587215912 (625h24m2.458298957s) [noise=true , address=145.79.227.32:4835 , piece_id=ZUMYIO55NMELJXSHOJSDXGEMGFEXYTAOFTQUCAWL2CW4UFZFRHMQ, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.648133958 (625h24m2.519217003s) [noise=true , address=145.79.227.41:4087 , piece_id=2NMUN4BSMAGHN4VFZM3AEQWHZN6S2IGMKSURL5EKEPB4BIIUAZFA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.651541286 (625h24m2.522624331s) [noise=true , address=145.79.227.39:4015 , piece_id=E3NXLXBM3IIK7ICGMIOGAZOGUVE24U4O3S5TLV5OFJAGD6NRGJVA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.708542554 (625h24m2.579625599s) [noise=true , address=145.79.227.38:4978 , piece_id=BOVSXWQR7323NSG3D2T2RNWG25TYTRQQ7C4RCP7RVEOM65HXFPLQ, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.724187364 (625h24m2.595270409s) [noise=true , address=145.79.227.42:4123 , piece_id=2KOMXT4FVOLDWYLFFTOT3Z7QRC2SQPUMY3WJROPBAS6EV7FNSTIA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.726115565 (625h24m2.5971986s) [noise=true , address=77.90.31.14:4663 , piece_id=HVYJJTDKP2PNFD4EPFN3NV5O3EPMD5REU3VF5KPAOZRMSSL3REQQ, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.897950584 (625h24m2.769033629s) [noise=true , address=45.91.116.10:4082 , piece_id=72ZVQSHOUQHKQUBCGG3UZDQWLLDSTCHQR7RRZYARVWSMVVGWPEYA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.988832015 (625h24m2.85991506s) [noise=true , address=45.91.116.7:4012 , piece_id=NZYM32SUJY5H5FVGJQL3ZBQZNGVJVBQMU7O4SRZX6RXPBK34NRFA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.988854056 (625h24m2.859937101s) [seg_pos={0 0}, upload=548018]: Segment reached optimal threshold of 13 pieces.
Dec 31 16:53:02.989090487 (625h24m2.860173532s) [noise=true , address=45.91.116.11:4118 , piece_id=2J6BXGMDT4EHSNS5IPGT4RUOFKFLJ56OFV4HUJUJA2SBYJ3JBK2Q, seg_pos={0 0}, upload=548018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:43678->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:43678->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43678->45.91.116.11:4118: use of closed network connection
Dec 31 16:53:02.989097470 (625h24m2.860180515s) [noise=true , address=45.91.116.11:4118 , piece_id=2J6BXGMDT4EHSNS5IPGT4RUOFKFLJ56OFV4HUJUJA2SBYJ3JBK2Q, seg_pos={0 0}, upload=548018]: Long tail cancellation due to optimal threshold
Dec 31 16:53:02.989284146 (625h24m2.860367191s) [noise=true , address=45.91.116.13:4154 , piece_id=HUUYW6JZ6RQHLX6K5QOPNOLA4WHR2QJZBLF6XAU2KG2GHIIXU67A, seg_pos={0 0}, upload=548018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 31 16:53:02.989301869 (625h24m2.860384914s) [noise=true , address=45.91.116.13:4154 , piece_id=HUUYW6JZ6RQHLX6K5QOPNOLA4WHR2QJZBLF6XAU2KG2GHIIXU67A, seg_pos={0 0}, upload=548018]: Long tail cancellation due to optimal threshold
Dec 31 16:53:02.989316708 (625h24m2.860399743s) [noise=true , address=45.91.116.9:4261 , piece_id=DKGELPLYLSA46I6OHE4K3NNLIJLRB66DNFCQK7HVYIPP54L5RZGQ, seg_pos={0 0}, upload=548018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:34968->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:34968->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34968->45.91.116.9:4261: use of closed network connection
Dec 31 16:53:02.989324032 (625h24m2.860407077s) [noise=true , address=45.91.116.9:4261 , piece_id=DKGELPLYLSA46I6OHE4K3NNLIJLRB66DNFCQK7HVYIPP54L5RZGQ, seg_pos={0 0}, upload=548018]: Long tail cancellation due to optimal threshold
Dec 31 16:53:02.989342877 (625h24m2.860425922s) [seg_pos={0 0}, upload=548018]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 16:53:02.989422960 (625h24m2.860506015s) [upload=548018]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:53:02.989428981 (625h24m2.860512037s) [upload=548018]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:53:03.276557574 (625h24m3.147640619s) [upload=548018]: Done uploading segments...
Dec 31 16:53:03.276584946 (625h24m3.147667991s) [upload=548018]: Upload finished. err: <nil>
Duration: 8.095252351s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.XX..XX........X....X.X...X....X.X.......X.XX..X..X...X.X.XX] duration:
30.003693707s trace:
https://jaeger.dev.storj.io/trace/5cfa15b3395c99c7
FAIL (2025-12-31 16:53:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: HVYJJTDKP2PNFD4EPFN3NV5O3EPMD5REU3VF5KPAOZRMSSL3REQQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: NZYM32SUJY5H5FVGJQL3ZBQZNGVJVBQMU7O4SRZX6RXPBK34NRFA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 72ZVQSHOUQHKQUBCGG3UZDQWLLDSTCHQR7RRZYARVWSMVVGWPEYA) context deadline exceeded
======LOGS======
(625h24m0.187935032s) [upload=548018]: Uploading segments...
Dec 31 16:53:00.317161005 (625h24m0.18824405s) [upload=548018]: Got next segment. Inline: false
Dec 31 16:53:00.317208526 (625h24m0.188291572s) [upload=548018]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 16:53:00.317214959 (625h24m0.188298004s) [upload=548018]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 16:53:00.559124607 (625h24m0.430207652s) [seg_pos={0 0}, upload=548018]: Begin upload segment...
Dec 31 16:53:00.559295773 (625h24m0.430378808s) [seg_pos={0 0}, upload=548018]: Done begin upload segment.
Dec 31 16:53:00.559331241 (625h24m0.430414276s) [noise=true , address=77.90.31.14:4663 , piece_id=HVYJJTDKP2PNFD4EPFN3NV5O3EPMD5REU3VF5KPAOZRMSSL3REQQ, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.560814085 (625h24m0.43189712s) [noise=true , address=145.79.227.22:4511 , piece_id=COUKMWSBDOUPUV6IBMTMM7MDRLJFR2QVHI4P6TD6K7XXLDKTID5Q, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.561408707 (625h24m0.432491752s) [noise=true , address=145.79.227.8:4330 , piece_id=XG67YYMPMX3ICYAE7AGFG4EFFIQILXFPEL563MCXVXCFX243LKBQ, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.561720671 (625h24m0.432803716s) [noise=true , address=145.79.227.41:4087 , piece_id=2NMUN4BSMAGHN4VFZM3AEQWHZN6S2IGMKSURL5EKEPB4BIIUAZFA, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.561960947 (625h24m0.433043982s) [noise=true , address=145.79.227.52:4446 , piece_id=RQCOSCULWHAOXN662DD7G5OYXFWYWF7K6ZASKM3DFJO35WTGBWAA, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.562345741 (625h24m0.433428786s) [noise=true , address=45.91.116.7:4012 , piece_id=NZYM32SUJY5H5FVGJQL3ZBQZNGVJVBQMU7O4SRZX6RXPBK34NRFA, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.562382371 (625h24m0.433465416s) [noise=true , address=45.91.116.11:4118 , piece_id=2J6BXGMDT4EHSNS5IPGT4RUOFKFLJ56OFV4HUJUJA2SBYJ3JBK2Q, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.562671952 (625h24m0.433754997s) [noise=true , address=45.91.116.13:4154 , piece_id=HUUYW6JZ6RQHLX6K5QOPNOLA4WHR2QJZBLF6XAU2KG2GHIIXU67A, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.562940604 (625h24m0.434023649s) [noise=true , address=145.79.227.39:4015 , piece_id=E3NXLXBM3IIK7ICGMIOGAZOGUVE24U4O3S5TLV5OFJAGD6NRGJVA, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.562982724 (625h24m0.434065799s) [noise=true , address=45.91.116.10:4082 , piece_id=72ZVQSHOUQHKQUBCGG3UZDQWLLDSTCHQR7RRZYARVWSMVVGWPEYA, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.563555213 (625h24m0.434638258s) [noise=true , address=145.79.227.38:4978 , piece_id=BOVSXWQR7323NSG3D2T2RNWG25TYTRQQ7C4RCP7RVEOM65HXFPLQ, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.564216943 (625h24m0.435299988s) [noise=true , address=145.79.227.42:4123 , piece_id=2KOMXT4FVOLDWYLFFTOT3Z7QRC2SQPUMY3WJROPBAS6EV7FNSTIA, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.564915253 (625h24m0.435998288s) [noise=true , address=45.91.116.9:4261 , piece_id=DKGELPLYLSA46I6OHE4K3NNLIJLRB66DNFCQK7HVYIPP54L5RZGQ, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.565695830 (625h24m0.436778875s) [noise=true , address=145.79.227.32:4835 , piece_id=ZUMYIO55NMELJXSHOJSDXGEMGFEXYTAOFTQUCAWL2CW4UFZFRHMQ, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.565999679 (625h24m0.437082714s) [noise=true , address=145.79.227.3:4150 , piece_id=IICUES4EAAPXXEUZHMS5M742ATNEZ6ZH2L6343NBEFWKN5A4EDTA, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:00.566426211 (625h24m0.437509246s) [noise=true , address=145.79.227.20:4437 , piece_id=ZB5POSRWAFSXFQBHGP6BNWH7FV6SHBUNWPB5M2GBGHFEZ72P5P6A, seg_pos={0 0}, upload=548018]: Uploading piece...
Dec 31 16:53:02.269582063 (625h24m2.140665109s) [upload=548018]: Next returned nil segment
Dec 31 16:53:02.269601401 (625h24m2.140684456s) [upload=548018]: Waiting for error group managing segments...
Dec 31 16:53:02.550702465 (625h24m2.42178551s) [noise=true , address=145.79.227.20:4437 , piece_id=ZB5POSRWAFSXFQBHGP6BNWH7FV6SHBUNWPB5M2GBGHFEZ72P5P6A, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.554501710 (625h24m2.425584765s) [noise=true , address=145.79.227.52:4446 , piece_id=RQCOSCULWHAOXN662DD7G5OYXFWYWF7K6ZASKM3DFJO35WTGBWAA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.569282905 (625h24m2.44036595s) [noise=true , address=145.79.227.3:4150 , piece_id=IICUES4EAAPXXEUZHMS5M742ATNEZ6ZH2L6343NBEFWKN5A4EDTA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.572389480 (625h24m2.443472526s) [noise=true , address=145.79.227.22:4511 , piece_id=COUKMWSBDOUPUV6IBMTMM7MDRLJFR2QVHI4P6TD6K7XXLDKTID5Q, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.579231618 (625h24m2.450314664s) [noise=true , address=145.79.227.8:4330 , piece_id=XG67YYMPMX3ICYAE7AGFG4EFFIQILXFPEL563MCXVXCFX243LKBQ, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.587215912 (625h24m2.458298957s) [noise=true , address=145.79.227.32:4835 , piece_id=ZUMYIO55NMELJXSHOJSDXGEMGFEXYTAOFTQUCAWL2CW4UFZFRHMQ, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.648133958 (625h24m2.519217003s) [noise=true , address=145.79.227.41:4087 , piece_id=2NMUN4BSMAGHN4VFZM3AEQWHZN6S2IGMKSURL5EKEPB4BIIUAZFA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.651541286 (625h24m2.522624331s) [noise=true , address=145.79.227.39:4015 , piece_id=E3NXLXBM3IIK7ICGMIOGAZOGUVE24U4O3S5TLV5OFJAGD6NRGJVA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.708542554 (625h24m2.579625599s) [noise=true , address=145.79.227.38:4978 , piece_id=BOVSXWQR7323NSG3D2T2RNWG25TYTRQQ7C4RCP7RVEOM65HXFPLQ, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.724187364 (625h24m2.595270409s) [noise=true , address=145.79.227.42:4123 , piece_id=2KOMXT4FVOLDWYLFFTOT3Z7QRC2SQPUMY3WJROPBAS6EV7FNSTIA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.726115565 (625h24m2.5971986s) [noise=true , address=77.90.31.14:4663 , piece_id=HVYJJTDKP2PNFD4EPFN3NV5O3EPMD5REU3VF5KPAOZRMSSL3REQQ, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.897950584 (625h24m2.769033629s) [noise=true , address=45.91.116.10:4082 , piece_id=72ZVQSHOUQHKQUBCGG3UZDQWLLDSTCHQR7RRZYARVWSMVVGWPEYA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.988832015 (625h24m2.85991506s) [noise=true , address=45.91.116.7:4012 , piece_id=NZYM32SUJY5H5FVGJQL3ZBQZNGVJVBQMU7O4SRZX6RXPBK34NRFA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil>
Dec 31 16:53:02.988854056 (625h24m2.859937101s) [seg_pos={0 0}, upload=548018]: Segment reached optimal threshold of 13 pieces.
Dec 31 16:53:02.989090487 (625h24m2.860173532s) [noise=true , address=45.91.116.11:4118 , piece_id=2J6BXGMDT4EHSNS5IPGT4RUOFKFLJ56OFV4HUJUJA2SBYJ3JBK2Q, seg_pos={0 0}, upload=548018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:43678->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:43678->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43678->45.91.116.11:4118: use of closed network connection
Dec 31 16:53:02.989097470 (625h24m2.860180515s) [noise=true , address=45.91.116.11:4118 , piece_id=2J6BXGMDT4EHSNS5IPGT4RUOFKFLJ56OFV4HUJUJA2SBYJ3JBK2Q, seg_pos={0 0}, upload=548018]: Long tail cancellation due to optimal threshold
Dec 31 16:53:02.989284146 (625h24m2.860367191s) [noise=true , address=45.91.116.13:4154 , piece_id=HUUYW6JZ6RQHLX6K5QOPNOLA4WHR2QJZBLF6XAU2KG2GHIIXU67A, seg_pos={0 0}, upload=548018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 31 16:53:02.989301869 (625h24m2.860384914s) [noise=true , address=45.91.116.13:4154 , piece_id=HUUYW6JZ6RQHLX6K5QOPNOLA4WHR2QJZBLF6XAU2KG2GHIIXU67A, seg_pos={0 0}, upload=548018]: Long tail cancellation due to optimal threshold
Dec 31 16:53:02.989316708 (625h24m2.860399743s) [noise=true , address=45.91.116.9:4261 , piece_id=DKGELPLYLSA46I6OHE4K3NNLIJLRB66DNFCQK7HVYIPP54L5RZGQ, seg_pos={0 0}, upload=548018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:34968->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:34968->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34968->45.91.116.9:4261: use of closed network connection
Dec 31 16:53:02.989324032 (625h24m2.860407077s) [noise=true , address=45.91.116.9:4261 , piece_id=DKGELPLYLSA46I6OHE4K3NNLIJLRB66DNFCQK7HVYIPP54L5RZGQ, seg_pos={0 0}, upload=548018]: Long tail cancellation due to optimal threshold
Dec 31 16:53:02.989342877 (625h24m2.860425922s) [seg_pos={0 0}, upload=548018]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 16:53:02.989422960 (625h24m2.860506015s) [upload=548018]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:53:02.989428981 (625h24m2.860512037s) [upload=548018]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:53:03.276557574 (625h24m3.147640619s) [upload=548018]: Done uploading segments...
Dec 31 16:53:03.276584946 (625h24m3.147667991s) [upload=548018]: Upload finished. err: <nil>
Duration: 30.003720928s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [........X..........X.....................................X..] duration:
33.657750366s trace:
https://jaeger.dev.storj.io/trace/1fe9f44dea5d609c
FAIL (2025-12-31 16:53:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
.241246563 (625h22m16.112329598s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.241981543 (625h22m16.113064588s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242111701 (625h22m16.113194746s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242433975 (625h22m16.11351702s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242713457 (625h22m16.113796492s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242802286 (625h22m16.113885331s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.243245350 (625h22m16.114328395s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.243631865 (625h22m16.114714901s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.244008643 (625h22m16.115091678s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.244381102 (625h22m16.115464147s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.244758581 (625h22m16.115841616s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.245176256 (625h22m16.116259301s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.245538285 (625h22m16.11662133s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.245960649 (625h22m16.117043684s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.246337136 (625h22m16.117420181s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.247466537 (625h22m16.118549572s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.247996927 (625h22m16.119079962s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:17.693671241 (625h22m17.564754286s) [upload=547992]: Next returned nil segment
Dec 31 16:51:17.693714864 (625h22m17.564797899s) [upload=547992]: Waiting for error group managing segments...
Dec 31 16:51:18.120111632 (625h22m17.991194667s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.122023212 (625h22m17.993106257s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.201357746 (625h22m18.072440781s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.293128900 (625h22m18.164211965s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.306933417 (625h22m18.178016452s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.357131932 (625h22m18.228214977s) [noise=true , address=77.90.31.11:4563 , piece_id=2LRC66FCB4DTXU44NFN5O5L6ZMGDCQ6LHHB7TLYJ2LZSYKRWJEYQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.390427070 (625h22m18.261510105s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.469358907 (625h22m18.340441942s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.675586202 (625h22m18.546669267s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.834344079 (625h22m18.705427114s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.845713768 (625h22m18.716796813s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.980140516 (625h22m18.851223551s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.980522623 (625h22m18.851605658s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.997884583 (625h22m18.868967618s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.997904631 (625h22m18.868987666s) [seg_pos={0 0}, upload=547992]: Segment reached optimal threshold of 14 pieces.
Dec 31 16:51:18.998092839 (625h22m18.869175874s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998099372 (625h22m18.869182417s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998204202 (625h22m18.869287237s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998210183 (625h22m18.869293218s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998213039 (625h22m18.869296074s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998221765 (625h22m18.86930481s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998312208 (625h22m18.869395243s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122S3PRcJwYfK5UEWESnYA7aTPHA6zCLFyBuFSm2Pb5wFSxDPiR): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998318149 (625h22m18.869401184s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998356041 (625h22m18.869439076s) [seg_pos={0 0}, upload=547992]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 16:51:18.998428418 (625h22m18.869511454s) [upload=547992]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:51:18.998433228 (625h22m18.869516273s) [upload=547992]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:51:19.285646870 (625h22m19.156729935s) [upload=547992]: Done uploading segments...
Dec 31 16:51:19.285688400 (625h22m19.156771445s) [upload=547992]: Upload finished. err: <nil>
Duration: 33.657777237s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [X...X....XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX...] duration:
3.607174747s trace:
https://jaeger.dev.storj.io/trace/19c6aa42963b3c0f
FAIL (2025-12-31 16:53:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.60719771s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
6.510578118s trace:
https://jaeger.dev.storj.io/trace/5582e72abe7a8819
FAIL (2025-12-31 16:53:05 UTC)
orbiter warning: context deadline exceeded
Duration: 6.510598367s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........X..........X.....................................X.] duration:
30.996530897s trace:
https://jaeger.dev.storj.io/trace/1fe9f44dea5d609c
FAIL (2025-12-31 16:52:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
.241246563 (625h22m16.112329598s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.241981543 (625h22m16.113064588s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242111701 (625h22m16.113194746s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242433975 (625h22m16.11351702s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242713457 (625h22m16.113796492s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242802286 (625h22m16.113885331s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.243245350 (625h22m16.114328395s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.243631865 (625h22m16.114714901s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.244008643 (625h22m16.115091678s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.244381102 (625h22m16.115464147s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.244758581 (625h22m16.115841616s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.245176256 (625h22m16.116259301s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.245538285 (625h22m16.11662133s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.245960649 (625h22m16.117043684s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.246337136 (625h22m16.117420181s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.247466537 (625h22m16.118549572s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.247996927 (625h22m16.119079962s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:17.693671241 (625h22m17.564754286s) [upload=547992]: Next returned nil segment
Dec 31 16:51:17.693714864 (625h22m17.564797899s) [upload=547992]: Waiting for error group managing segments...
Dec 31 16:51:18.120111632 (625h22m17.991194667s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.122023212 (625h22m17.993106257s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.201357746 (625h22m18.072440781s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.293128900 (625h22m18.164211965s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.306933417 (625h22m18.178016452s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.357131932 (625h22m18.228214977s) [noise=true , address=77.90.31.11:4563 , piece_id=2LRC66FCB4DTXU44NFN5O5L6ZMGDCQ6LHHB7TLYJ2LZSYKRWJEYQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.390427070 (625h22m18.261510105s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.469358907 (625h22m18.340441942s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.675586202 (625h22m18.546669267s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.834344079 (625h22m18.705427114s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.845713768 (625h22m18.716796813s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.980140516 (625h22m18.851223551s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.980522623 (625h22m18.851605658s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.997884583 (625h22m18.868967618s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.997904631 (625h22m18.868987666s) [seg_pos={0 0}, upload=547992]: Segment reached optimal threshold of 14 pieces.
Dec 31 16:51:18.998092839 (625h22m18.869175874s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998099372 (625h22m18.869182417s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998204202 (625h22m18.869287237s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998210183 (625h22m18.869293218s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998213039 (625h22m18.869296074s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998221765 (625h22m18.86930481s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998312208 (625h22m18.869395243s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122S3PRcJwYfK5UEWESnYA7aTPHA6zCLFyBuFSm2Pb5wFSxDPiR): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998318149 (625h22m18.869401184s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998356041 (625h22m18.869439076s) [seg_pos={0 0}, upload=547992]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 16:51:18.998428418 (625h22m18.869511454s) [upload=547992]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:51:18.998433228 (625h22m18.869516273s) [upload=547992]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:51:19.285646870 (625h22m19.156729935s) [upload=547992]: Done uploading segments...
Dec 31 16:51:19.285688400 (625h22m19.156771445s) [upload=547992]: Upload finished. err: <nil>
Duration: 30.996561116s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [..XX..XX........X....X.X...X....X.X.......X.XX..X..X...X.X.X] duration:
30.002644045s trace:
https://jaeger.dev.storj.io/trace/3df2b37f6c869d10
FAIL (2025-12-31 16:52:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: RJ5LFCAVPJPYGDEQ5UE6GM2Y3BKZMAZUB6SCZB2FKQ6ZMXHXTAEQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: EOKR653TKNQIX2NOGXBKC2XYBNDNKTMSK352BDRQNPQ3QZGJE7ZA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: S27E2P53OPIFMITM45LLMRWF4YEJE3JHE3REC3VJB353SDHJ2YXQ) context deadline exceeded
======LOGS======
Dec 31 16:52:07.448440614 (625h23m7.319523649s) [upload=548003]: Starting upload
Dec 31 16:52:07.448534873 (625h23m7.319617918s) [upload=548003]: Done starting upload
Dec 31 16:52:07.448605697 (625h23m7.319688732s) [upload=548003]: Uploading segments...
Dec 31 16:52:07.448830025 (625h23m7.31991307s) [upload=548003]: Got next segment. Inline: false
Dec 31 16:52:07.449025136 (625h23m7.320108171s) [upload=548003]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 16:52:07.449033202 (625h23m7.320116247s) [upload=548003]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 16:52:07.696415663 (625h23m7.567498708s) [seg_pos={0 0}, upload=548003]: Begin upload segment...
Dec 31 16:52:07.696478633 (625h23m7.567561678s) [seg_pos={0 0}, upload=548003]: Done begin upload segment.
Dec 31 16:52:07.696500764 (625h23m7.567583809s) [noise=true , address=145.79.227.20:4438 , piece_id=J3OGLW3BAROIWAQVC36W47NB5UIYR44Y57BHS7S5KAVVYIXKES3Q, seg_pos={0 0}, upload=548003]: Uploading piece...
Dec 31 16:52:07.696522135 (625h23m7.56760517s) [noise=true , address=45.91.116.14:4192 , piece_id=JPDOMSEQNR4676DVMOPZPRUEM2RBW55BJJHEJJ4HIRYE7FLS7E7A, seg_pos={0 0}, upload=548003]: Uploading piece...
Dec 31 16:52:07.697181461 (625h23m7.568264506s) [noise=true , address=45.91.116.9:4263 , piece_id=S27E2P53OPIFMITM45LLMRWF4YEJE3JHE3REC3VJB353SDHJ2YXQ, seg_pos={0 0}, upload=548003]: Uploading piece...
Dec 31 16:52:07.697585610 (625h23m7.568668645s) [noise=true , address=77.90.31.21:4880 , piece_id=RJ5LFCAVPJPYGDEQ5UE6GM2Y3BKZMAZUB6SCZB2FKQ6ZMXHXTAEQ, seg_pos={0 0}, upload=548003]: Uploading piece...
Dec 31 16:52:07.697991964 (625h23m7.569075009s) [noise=true , address=145.79.227.48:4303 , piece_id=M76HKAHB66KPBIHSQU6YLE3UEZT2FDASTZOOQTHELVW7O42ED5TA, seg_pos={0 0}, upload=548003]: Uploading piece...
Dec 31 16:52:07.698363962 (625h23m7.569447008s) [noise=true , address=145.79.227.10:4401 , piece_id=VY5VYDQ27J6GZT2DJMY2BWASD3SN6F3JOMTLACLUWWNTT7WWSZPA, seg_pos={0 0}, upload=548003]: Uploading piece...
Dec 31 16:52:07.698727164 (625h23m7.569810209s) [noise=true , address=145.79.227.43:4158 , piece_id=MKTWCS7FKV6FJSKTEPT7OCBJNZPD6ZGM6LGY7Q3J3MUPXQGRKVOQ, seg_pos={0 0}, upload=548003]: Uploading piece...
Dec 31 16:52:07.699132686 (625h23m7.570215731s) [noise=true , address=77.90.31.17:4773 , piece_id=LSRB5SXJEOV7RQXKXKDXZ5TP5H36LTFGXMJBGVSH6PIYGDAETKTQ, seg_pos={0 0}, upload=548003]: Uploading piece...
Dec 31 16:52:07.699581771 (625h23m7.570664806s) [noise=true , address=145.79.227.25:4619 , piece_id=TSG6P6HGSDQ3SJ762SXTXKTUQE2RKV5MW7LX77V6P3IWV6ED3C3A, seg_pos={0 0}, upload=548003]: Uploading piece...
Dec 31 16:52:07.700034804 (625h23m7.571117849s) [noise=true , address=145.79.227.52:4445 , piece_id=NA3LOZY6RHDC45GI76UXXM4F6QN53XRNR3UVVRAHYVHMLHQ4EUMQ, seg_pos={0 0}, upload=548003]: Uploading piece...
Dec 31 16:52:07.701074304 (625h23m7.572157339s) [noise=true , address=145.79.227.2:4115 , piece_id=5T5PKDZ6MUKWQ7C54A26HESEDEZ6PYD6YSGSCEWK7R2AFDKLUYMQ, seg_pos={0 0}, upload=548003]: Uploading piece...
Dec 31 16:52:07.701180105 (625h23m7.57226314s) [noise=true , address=145.79.227.16:4912 , piece_id=5G3PEN6VZTWDG36SDVFWB6IH2KRTT4M4WXMOUIM65NXZKQW4VZYA, seg_pos={0 0}, upload=548003]: Uploading piece...
Dec 31 16:52:07.702344933 (625h23m7.573427978s) [noise=true , address=145.79.227.27:4691 , piece_id=CBSV5SVEI4G2QRC6YRVVBUX27TBMIK7DCSBELL2WMZFMRCXO662Q, seg_pos={0 0}, upload=548003]: Uploading piece...
Dec 31 16:52:07.702614065 (625h23m7.57369711s) [noise=true , address=45.91.116.12:4228 , piece_id=ASSF452HMF3XCCXWXZP3HQN5SM7ORYWVNAXAW4XLR2M62HPW2GQQ, seg_pos={0 0}, upload=548003]: Uploading piece...
Dec 31 16:52:07.703533977 (625h23m7.574617012s) [noise=true , address=145.79.227.9:4367 , piece_id=JHVU6SSS7BZ5BFI2O5VLYHAU6RVXS7NMKQISS2465N7UIVDMWI5Q, seg_pos={0 0}, upload=548003]: Uploading piece...
Dec 31 16:52:07.703906065 (625h23m7.57498911s) [noise=true , address=45.91.116.6:4973 , piece_id=EOKR653TKNQIX2NOGXBKC2XYBNDNKTMSK352BDRQNPQ3QZGJE7ZA, seg_pos={0 0}, upload=548003]: Uploading piece...
Dec 31 16:52:09.392753555 (625h23m9.2638366s) [upload=548003]: Next returned nil segment
Dec 31 16:52:09.392793131 (625h23m9.263876176s) [upload=548003]: Waiting for error group managing segments...
Dec 31 16:52:09.671552467 (625h23m9.542635502s) [noise=true , address=145.79.227.10:4401 , piece_id=VY5VYDQ27J6GZT2DJMY2BWASD3SN6F3JOMTLACLUWWNTT7WWSZPA, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil>
Dec 31 16:52:09.688655704 (625h23m9.559738749s) [noise=true , address=145.79.227.25:4619 , piece_id=TSG6P6HGSDQ3SJ762SXTXKTUQE2RKV5MW7LX77V6P3IWV6ED3C3A, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil>
Dec 31 16:52:09.693405949 (625h23m9.564488984s) [noise=true , address=145.79.227.20:4438 , piece_id=J3OGLW3BAROIWAQVC36W47NB5UIYR44Y57BHS7S5KAVVYIXKES3Q, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil>
Dec 31 16:52:09.699196074 (625h23m9.570279119s) [noise=true , address=145.79.227.27:4691 , piece_id=CBSV5SVEI4G2QRC6YRVVBUX27TBMIK7DCSBELL2WMZFMRCXO662Q, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil>
Dec 31 16:52:09.702592241 (625h23m9.573675286s) [noise=true , address=145.79.227.52:4445 , piece_id=NA3LOZY6RHDC45GI76UXXM4F6QN53XRNR3UVVRAHYVHMLHQ4EUMQ, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil>
Dec 31 16:52:09.710118462 (625h23m9.581201507s) [noise=true , address=145.79.227.48:4303 , piece_id=M76HKAHB66KPBIHSQU6YLE3UEZT2FDASTZOOQTHELVW7O42ED5TA, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil>
Dec 31 16:52:09.717270661 (625h23m9.588353696s) [noise=true , address=145.79.227.16:4912 , piece_id=5G3PEN6VZTWDG36SDVFWB6IH2KRTT4M4WXMOUIM65NXZKQW4VZYA, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil>
Dec 31 16:52:09.762149617 (625h23m9.633232662s) [noise=true , address=145.79.227.9:4367 , piece_id=JHVU6SSS7BZ5BFI2O5VLYHAU6RVXS7NMKQISS2465N7UIVDMWI5Q, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil>
Dec 31 16:52:09.961121327 (625h23m9.832204372s) [noise=true , address=77.90.31.21:4880 , piece_id=RJ5LFCAVPJPYGDEQ5UE6GM2Y3BKZMAZUB6SCZB2FKQ6ZMXHXTAEQ, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil>
Dec 31 16:52:10.106798796 (625h23m9.977881831s) [noise=true , address=45.91.116.12:4228 , piece_id=ASSF452HMF3XCCXWXZP3HQN5SM7ORYWVNAXAW4XLR2M62HPW2GQQ, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil>
Dec 31 16:52:10.144953117 (625h23m10.016036152s) [noise=true , address=45.91.116.14:4192 , piece_id=JPDOMSEQNR4676DVMOPZPRUEM2RBW55BJJHEJJ4HIRYE7FLS7E7A, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil>
Dec 31 16:52:10.301599614 (625h23m10.172682649s) [noise=true , address=45.91.116.6:4973 , piece_id=EOKR653TKNQIX2NOGXBKC2XYBNDNKTMSK352BDRQNPQ3QZGJE7ZA, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil>
Dec 31 16:52:10.582860483 (625h23m10.453943518s) [noise=true , address=45.91.116.9:4263 , piece_id=S27E2P53OPIFMITM45LLMRWF4YEJE3JHE3REC3VJB353SDHJ2YXQ, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil>
Dec 31 16:52:10.582885370 (625h23m10.453968405s) [seg_pos={0 0}, upload=548003]: Segment reached optimal threshold of 13 pieces.
Dec 31 16:52:10.583123755 (625h23m10.4542068s) [noise=true , address=145.79.227.43:4158 , piece_id=MKTWCS7FKV6FJSKTEPT7OCBJNZPD6ZGM6LGY7Q3J3MUPXQGRKVOQ, seg_pos={0 0}, upload=548003]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF
Dec 31 16:52:10.583132952 (625h23m10.454215997s) [noise=true , address=145.79.227.43:4158 , piece_id=MKTWCS7FKV6FJSKTEPT7OCBJNZPD6ZGM6LGY7Q3J3MUPXQGRKVOQ, seg_pos={0 0}, upload=548003]: Long tail cancellation due to optimal threshold
Dec 31 16:52:10.583320418 (625h23m10.454403463s) [noise=true , address=145.79.227.2:4115 , piece_id=5T5PKDZ6MUKWQ7C54A26HESEDEZ6PYD6YSGSCEWK7R2AFDKLUYMQ, seg_pos={0 0}, upload=548003]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:42976->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:42976->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42976->145.79.227.2:4115: use of closed network connection
Dec 31 16:52:10.583339255 (625h23m10.4544223s) [noise=true , address=145.79.227.2:4115 , piece_id=5T5PKDZ6MUKWQ7C54A26HESEDEZ6PYD6YSGSCEWK7R2AFDKLUYMQ, seg_pos={0 0}, upload=548003]: Long tail cancellation due to optimal threshold
Dec 31 16:52:10.583448493 (625h23m10.454531538s) [noise=true , address=77.90.31.17:4773 , piece_id=LSRB5SXJEOV7RQXKXKDXZ5TP5H36LTFGXMJBGVSH6PIYGDAETKTQ, seg_pos={0 0}, upload=548003]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 31 16:52:10.583458581 (625h23m10.454541627s) [noise=true , address=77.90.31.17:4773 , piece_id=LSRB5SXJEOV7RQXKXKDXZ5TP5H36LTFGXMJBGVSH6PIYGDAETKTQ, seg_pos={0 0}, upload=548003]: Long tail cancellation due to optimal threshold
Dec 31 16:52:10.583485132 (625h23m10.454568177s) [seg_pos={0 0}, upload=548003]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 16:52:10.583567569 (625h23m10.454650604s) [upload=548003]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:52:10.583575395 (625h23m10.45465843s) [upload=548003]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:52:10.880724708 (625h23m10.751807763s) [upload=548003]: Done uploading segments...
Dec 31 16:52:10.880748953 (625h23m10.751831999s) [upload=548003]: Upload finished. err: <nil>
Duration: 30.002668311s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [.X...X....XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX..] duration:
4.164442846s trace:
https://jaeger.dev.storj.io/trace/19c6aa42963b3c0f
FAIL (2025-12-31 16:52:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.164470418s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........X..........X.....................................X] duration:
1m0.002952776s trace:
https://jaeger.dev.storj.io/trace/1fe9f44dea5d609c
FAIL (2025-12-31 16:52:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
.241246563 (625h22m16.112329598s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.241981543 (625h22m16.113064588s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242111701 (625h22m16.113194746s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242433975 (625h22m16.11351702s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242713457 (625h22m16.113796492s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.242802286 (625h22m16.113885331s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.243245350 (625h22m16.114328395s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.243631865 (625h22m16.114714901s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.244008643 (625h22m16.115091678s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.244381102 (625h22m16.115464147s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.244758581 (625h22m16.115841616s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.245176256 (625h22m16.116259301s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.245538285 (625h22m16.11662133s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.245960649 (625h22m16.117043684s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.246337136 (625h22m16.117420181s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.247466537 (625h22m16.118549572s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:16.247996927 (625h22m16.119079962s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Uploading piece...
Dec 31 16:51:17.693671241 (625h22m17.564754286s) [upload=547992]: Next returned nil segment
Dec 31 16:51:17.693714864 (625h22m17.564797899s) [upload=547992]: Waiting for error group managing segments...
Dec 31 16:51:18.120111632 (625h22m17.991194667s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.122023212 (625h22m17.993106257s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.201357746 (625h22m18.072440781s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.293128900 (625h22m18.164211965s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.306933417 (625h22m18.178016452s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.357131932 (625h22m18.228214977s) [noise=true , address=77.90.31.11:4563 , piece_id=2LRC66FCB4DTXU44NFN5O5L6ZMGDCQ6LHHB7TLYJ2LZSYKRWJEYQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.390427070 (625h22m18.261510105s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.469358907 (625h22m18.340441942s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.675586202 (625h22m18.546669267s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.834344079 (625h22m18.705427114s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.845713768 (625h22m18.716796813s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.980140516 (625h22m18.851223551s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.980522623 (625h22m18.851605658s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.997884583 (625h22m18.868967618s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil>
Dec 31 16:51:18.997904631 (625h22m18.868987666s) [seg_pos={0 0}, upload=547992]: Segment reached optimal threshold of 14 pieces.
Dec 31 16:51:18.998092839 (625h22m18.869175874s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998099372 (625h22m18.869182417s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998204202 (625h22m18.869287237s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998210183 (625h22m18.869293218s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998213039 (625h22m18.869296074s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998221765 (625h22m18.86930481s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998312208 (625h22m18.869395243s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122S3PRcJwYfK5UEWESnYA7aTPHA6zCLFyBuFSm2Pb5wFSxDPiR): protocol: expected piece hash; context canceled; EOF
Dec 31 16:51:18.998318149 (625h22m18.869401184s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold
Dec 31 16:51:18.998356041 (625h22m18.869439076s) [seg_pos={0 0}, upload=547992]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 16:51:18.998428418 (625h22m18.869511454s) [upload=547992]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:51:18.998433228 (625h22m18.869516273s) [upload=547992]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:51:19.285646870 (625h22m19.156729935s) [upload=547992]: Done uploading segments...
Dec 31 16:51:19.285688400 (625h22m19.156771445s) [upload=547992]: Upload finished. err: <nil>
Duration: 1m0.002962754s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
7.110370571s trace:
https://jaeger.dev.storj.io/trace/5582e72abe7a8819
FAIL (2025-12-31 16:52:07 UTC)
orbiter warning: context deadline exceeded
Duration: 7.110400457s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
6.226168017s trace:
https://jaeger.dev.storj.io/trace/5582e72abe7a8819
FAIL (2025-12-31 16:51:36 UTC)
orbiter warning: context deadline exceeded
Duration: 6.226197294s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [..X...X....XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX.] duration:
5.001581062s trace:
https://jaeger.dev.storj.io/trace/19c6aa42963b3c0f
FAIL (2025-12-31 16:51:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.0016066s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [...XX..XX........X....X.X...X....X.X.......X.XX..X..X...X.X.] duration:
6.106002454s trace:
https://jaeger.dev.storj.io/trace/2bd44400444d3a18
FAIL (2025-12-31 16:51:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: WPT5LIUIRYEBO2ZDDGNLPJ3NBTAR7OSOBI3SUG7QLVUDJ33NU55A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: MBK7ZDCWDBFVBPK7BK7YYMLRC3KJZBGE3TBXR2TYCBLAVFDHYJTQ) context deadline exceeded
======LOGS======
Dec 31 16:50:06.195526122 (625h21m6.066609167s) [upload=547974]: Starting upload
Dec 31 16:50:06.195619359 (625h21m6.066702394s) [upload=547974]: Done starting upload
Dec 31 16:50:06.195695003 (625h21m6.066778048s) [upload=547974]: Uploading segments...
Dec 31 16:50:06.196071881 (625h21m6.067154926s) [upload=547974]: Got next segment. Inline: false
Dec 31 16:50:06.196113159 (625h21m6.067196205s) [upload=547974]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 16:50:06.196118990 (625h21m6.067202035s) [upload=547974]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 16:50:06.443263899 (625h21m6.314346944s) [seg_pos={0 0}, upload=547974]: Begin upload segment...
Dec 31 16:50:06.443330886 (625h21m6.314413931s) [seg_pos={0 0}, upload=547974]: Done begin upload segment.
Dec 31 16:50:06.443374961 (625h21m6.314458016s) [noise=true , address=145.79.227.6:4257 , piece_id=T5FDGD74QDOUXLHJ4Q2EHOM5U3TLWBTFMY2U4B7FHW53RJNG6H3A, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.443405529 (625h21m6.314488564s) [noise=true , address=145.79.227.9:4367 , piece_id=APZKQVIBW66G4FHIJPK2IGQZRLHMG7NKKK336HPHH6ZVK4454SMQ, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.444110832 (625h21m6.315193877s) [noise=true , address=77.90.31.12:4589 , piece_id=WPT5LIUIRYEBO2ZDDGNLPJ3NBTAR7OSOBI3SUG7QLVUDJ33NU55A, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.444640640 (625h21m6.315723685s) [noise=true , address=45.91.116.11:4120 , piece_id=SXXI2AWRCTQA2L25E2VKQPGARWWONK25WCKILETCOJH7Y6JAVO2A, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.445877787 (625h21m6.316960832s) [noise=true , address=145.79.227.3:4150 , piece_id=IC3HP2IF4TIG6XG7TCGD6IJO264QIP46YIMPJOYHC7J3FPZJUMKA, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.446447671 (625h21m6.317530716s) [noise=true , address=145.79.227.21:4474 , piece_id=O62BG3LXFAZLZAM6KIYXVUC7I4WG3LCJUFX6S6JH4NWB24L2HHAQ, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.446917465 (625h21m6.31800051s) [noise=true , address=45.91.116.5:4300 , piece_id=MBK7ZDCWDBFVBPK7BK7YYMLRC3KJZBGE3TBXR2TYCBLAVFDHYJTQ, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.447282882 (625h21m6.318365927s) [noise=true , address=145.79.227.14:4839 , piece_id=NTG7XLUKRUHRV5GZSOLCZIHFFDUKLELOGQTSAQ6VX5P5JU4Z4WLA, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.447618731 (625h21m6.318701766s) [noise=true , address=145.79.227.29:4941 , piece_id=WSHSFOSXDXPRYR2PDPSXMPCP5VYCIM4GRIGWGVMKZKYDCIJVMGZQ, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.448009495 (625h21m6.319092541s) [noise=true , address=145.79.227.20:4437 , piece_id=Y5HPFVGLYUV7NZKX6N56GAIBE6HO3R4WF6WK27FR2HBZSK7KYOVQ, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.448452558 (625h21m6.319535603s) [noise=true , address=145.79.227.51:4410 , piece_id=J6V53T5NJKA4WR37XP4QHEOM3RPAKNG52YFMR5KEXWYTY2XDKZZA, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.448824156 (625h21m6.319907191s) [noise=true , address=145.79.227.50:4373 , piece_id=E3KLYBUJAMKV4S2UYRCVJ46YGD2VFR6WH2ERFVD6WTN2ZBC67NJA, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.449305343 (625h21m6.320388388s) [noise=true , address=145.79.227.45:4231 , piece_id=VDZQMBUR6WHXWABVHPLTVC5IJMPQYDW2X2XIDJCCHEZEIUGPDXPQ, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.449760189 (625h21m6.320843234s) [noise=true , address=77.90.31.13:4627 , piece_id=JC7S4L3N5RQG7MDCKGHF63UX6YNNWRNQN4EGMMB3XFGAXGEMBJQQ, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.450367615 (625h21m6.32145066s) [noise=true , address=45.91.116.6:4973 , piece_id=KOXRKS25AF6P2IYUHSJ5OEOQWZA3LNGDXM4EUSLPVJY57IESHRCQ, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.450818033 (625h21m6.321901068s) [noise=true , address=145.79.227.15:4876 , piece_id=6XVXA3CTQMTDUVSU6U2ZZPEH3Y6AQXDYKZMBHR6Q5NUYHYNL677Q, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:08.114790637 (625h21m7.985873672s) [upload=547974]: Next returned nil segment
Dec 31 16:50:08.114805166 (625h21m7.985888211s) [upload=547974]: Waiting for error group managing segments...
Dec 31 16:50:08.425962442 (625h21m8.297045477s) [noise=true , address=145.79.227.51:4410 , piece_id=J6V53T5NJKA4WR37XP4QHEOM3RPAKNG52YFMR5KEXWYTY2XDKZZA, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.427604178 (625h21m8.298687223s) [noise=true , address=145.79.227.9:4367 , piece_id=APZKQVIBW66G4FHIJPK2IGQZRLHMG7NKKK336HPHH6ZVK4454SMQ, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.435654106 (625h21m8.306737141s) [noise=true , address=145.79.227.3:4150 , piece_id=IC3HP2IF4TIG6XG7TCGD6IJO264QIP46YIMPJOYHC7J3FPZJUMKA, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.439789290 (625h21m8.310872325s) [noise=true , address=145.79.227.50:4373 , piece_id=E3KLYBUJAMKV4S2UYRCVJ46YGD2VFR6WH2ERFVD6WTN2ZBC67NJA, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.439893809 (625h21m8.310976854s) [noise=true , address=145.79.227.45:4231 , piece_id=VDZQMBUR6WHXWABVHPLTVC5IJMPQYDW2X2XIDJCCHEZEIUGPDXPQ, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.445041882 (625h21m8.316124957s) [noise=true , address=145.79.227.15:4876 , piece_id=6XVXA3CTQMTDUVSU6U2ZZPEH3Y6AQXDYKZMBHR6Q5NUYHYNL677Q, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.450778004 (625h21m8.32186105s) [noise=true , address=145.79.227.6:4257 , piece_id=T5FDGD74QDOUXLHJ4Q2EHOM5U3TLWBTFMY2U4B7FHW53RJNG6H3A, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.452441742 (625h21m8.323524787s) [noise=true , address=145.79.227.20:4437 , piece_id=Y5HPFVGLYUV7NZKX6N56GAIBE6HO3R4WF6WK27FR2HBZSK7KYOVQ, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.459060124 (625h21m8.330143179s) [noise=true , address=145.79.227.21:4474 , piece_id=O62BG3LXFAZLZAM6KIYXVUC7I4WG3LCJUFX6S6JH4NWB24L2HHAQ, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.493893091 (625h21m8.364976137s) [noise=true , address=145.79.227.14:4839 , piece_id=NTG7XLUKRUHRV5GZSOLCZIHFFDUKLELOGQTSAQ6VX5P5JU4Z4WLA, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.630734005 (625h21m8.50181705s) [noise=true , address=77.90.31.12:4589 , piece_id=WPT5LIUIRYEBO2ZDDGNLPJ3NBTAR7OSOBI3SUG7QLVUDJ33NU55A, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.792135947 (625h21m8.663218992s) [noise=true , address=45.91.116.6:4973 , piece_id=KOXRKS25AF6P2IYUHSJ5OEOQWZA3LNGDXM4EUSLPVJY57IESHRCQ, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.919533560 (625h21m8.790616595s) [noise=true , address=45.91.116.5:4300 , piece_id=MBK7ZDCWDBFVBPK7BK7YYMLRC3KJZBGE3TBXR2TYCBLAVFDHYJTQ, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.919556524 (625h21m8.790639569s) [seg_pos={0 0}, upload=547974]: Segment reached optimal threshold of 13 pieces.
Dec 31 16:50:08.919771343 (625h21m8.790854388s) [noise=true , address=145.79.227.29:4941 , piece_id=WSHSFOSXDXPRYR2PDPSXMPCP5VYCIM4GRIGWGVMKZKYDCIJVMGZQ, seg_pos={0 0}, upload=547974]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 31 16:50:08.919778897 (625h21m8.790861942s) [noise=true , address=145.79.227.29:4941 , piece_id=WSHSFOSXDXPRYR2PDPSXMPCP5VYCIM4GRIGWGVMKZKYDCIJVMGZQ, seg_pos={0 0}, upload=547974]: Long tail cancellation due to optimal threshold
Dec 31 16:50:08.920124055 (625h21m8.7912071s) [noise=true , address=77.90.31.13:4627 , piece_id=JC7S4L3N5RQG7MDCKGHF63UX6YNNWRNQN4EGMMB3XFGAXGEMBJQQ, seg_pos={0 0}, upload=547974]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: read tcp 188.166.225.170:54978->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 16:50:08.920131669 (625h21m8.791214704s) [noise=true , address=77.90.31.13:4627 , piece_id=JC7S4L3N5RQG7MDCKGHF63UX6YNNWRNQN4EGMMB3XFGAXGEMBJQQ, seg_pos={0 0}, upload=547974]: Long tail cancellation due to optimal threshold
Dec 31 16:50:08.920151146 (625h21m8.791234202s) [noise=true , address=45.91.116.11:4120 , piece_id=SXXI2AWRCTQA2L25E2VKQPGARWWONK25WCKILETCOJH7Y6JAVO2A, seg_pos={0 0}, upload=547974]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: read tcp 188.166.225.170:49932->45.91.116.11:4120: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 16:50:08.920171414 (625h21m8.791254449s) [noise=true , address=45.91.116.11:4120 , piece_id=SXXI2AWRCTQA2L25E2VKQPGARWWONK25WCKILETCOJH7Y6JAVO2A, seg_pos={0 0}, upload=547974]: Long tail cancellation due to optimal threshold
Dec 31 16:50:08.920200039 (625h21m8.791283084s) [seg_pos={0 0}, upload=547974]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 16:50:08.920314899 (625h21m8.791397944s) [upload=547974]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:50:08.920322423 (625h21m8.791405468s) [upload=547974]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:50:09.213054485 (625h21m9.08413752s) [upload=547974]: Done uploading segments...
Dec 31 16:50:09.213076607 (625h21m9.084159652s) [upload=547974]: Upload finished. err: <nil>
Duration: 6.106021671s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........X..........X.....................................] duration:
48.511754364s trace:
https://jaeger.dev.storj.io/trace/5b4dfa890c7fc19f
FAIL (2025-12-31 16:51:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
QCFUNRQ, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.048997877 (624h48m14.920080912s) [noise=true , address=77.90.31.13:4657 , piece_id=5ITVL3WNXVLJLNGPJS5YSLTOSLULDQGK6F64TJQ6K52MZG2UBIMQ, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.049460398 (624h48m14.920543463s) [noise=true , address=145.79.227.6:4278 , piece_id=UPW54EA7CL2QDHHSBOCOUHOJ4PO664J7HPFRMKDZVRJWLQLZQE3Q, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.049958216 (624h48m14.921041251s) [noise=true , address=77.90.31.12:4606 , piece_id=CMC465DGAG3VRLPTRWL2TNC6MD3HFPIABJJQUTHVSL32LFWPQRBA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.049996479 (624h48m14.921079514s) [noise=true , address=145.79.227.35:4492 , piece_id=5KJRZLQ3H3S6SFZWLTKRQU3JIN7RY4E5FJFDDMPHCY6LCYXTVOPA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.050251765 (624h48m14.9213348s) [noise=true , address=5.231.73.3:4030 , piece_id=II77J2BA2EEOJXS4KPGI3GTC4AYQ5Z6NL4ABJCGP2XZEFBP5ABOA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.050429022 (624h48m14.921512057s) [noise=true , address=45.91.116.9:4271 , piece_id=VZJVLEOMS452TXPJOMIVF4K36YRPLQW3EGZNMEB4BZ7HK7PTDQXQ, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.050521198 (624h48m14.921604233s) [noise=true , address=145.79.227.24:4614 , piece_id=6BJGK6OQWTHZCY3GQSQV7LVJWJOAJSFQSSVPDJ3RGX4EUFIISFRA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.050715528 (624h48m14.921798573s) [noise=true , address=145.79.227.23:4575 , piece_id=NXYLJMIMKJYMJ2UF5BYXYODDUOCROHYNDEHXCO2G5IFVVZQFLXYQ, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.050816169 (624h48m14.921899204s) [noise=true , address=145.79.227.38:4004 , piece_id=OMFDOFGQ3EBWZPON3I3DP6LWBP3BWWEEFHU34EXZBF2BAJLP3TNQ, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.050998136 (624h48m14.922081171s) [noise=true , address=145.79.227.17:4985 , piece_id=Q37B2D24JXNSSJIEIBRSYEE6IEOQV7IG4SEIGLFWLW2PBTWZU5TA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.051085192 (624h48m14.922168227s) [noise=true , address=145.79.227.44:4219 , piece_id=TQUYQDIUJWUX43HZCUOHECBX77YNCL2SZ6BMGX2VQ4T2XUPWNKJA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.051462040 (624h48m14.922545075s) [noise=true , address=77.90.31.17:4788 , piece_id=4G5Z3ECGM5HBH7PELBGGDKXGDPHTNRHKEBXFEKVEP5ZDFXVIGWUA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.051871809 (624h48m14.922954844s) [noise=true , address=77.90.31.16:4760 , piece_id=P56LD3DX5YLE2MPV3RORGLX25G5YDT2TNOVCA5CLGGBYZN6LUUTQ, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.052116004 (624h48m14.923199039s) [noise=true , address=145.79.227.33:4894 , piece_id=MYHRMEIWJDASEUHHTGRKAN5IRNE7QBTO4VNX62LHJVGBIJS5UVLA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.052414814 (624h48m14.923497859s) [noise=true , address=145.79.227.19:4087 , piece_id=7EXAHR36NSDTHNUWQQZTIS46XWFQUWTKAEIPTMSQHZHDJHCKI6PA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:16.513009118 (624h48m16.384092153s) [upload=547498]: Next returned nil segment
Dec 31 16:17:16.513023495 (624h48m16.38410654s) [upload=547498]: Waiting for error group managing segments...
Dec 31 16:17:16.908124481 (624h48m16.779207526s) [noise=true , address=145.79.227.19:4087 , piece_id=7EXAHR36NSDTHNUWQQZTIS46XWFQUWTKAEIPTMSQHZHDJHCKI6PA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:16.918737279 (624h48m16.789820325s) [noise=true , address=145.79.227.6:4278 , piece_id=UPW54EA7CL2QDHHSBOCOUHOJ4PO664J7HPFRMKDZVRJWLQLZQE3Q, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:16.964651124 (624h48m16.835734159s) [noise=true , address=145.79.227.17:4985 , piece_id=Q37B2D24JXNSSJIEIBRSYEE6IEOQV7IG4SEIGLFWLW2PBTWZU5TA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:16.994270222 (624h48m16.865353258s) [noise=true , address=145.79.227.38:4004 , piece_id=OMFDOFGQ3EBWZPON3I3DP6LWBP3BWWEEFHU34EXZBF2BAJLP3TNQ, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:16.997634448 (624h48m16.868717483s) [noise=true , address=145.79.227.23:4575 , piece_id=NXYLJMIMKJYMJ2UF5BYXYODDUOCROHYNDEHXCO2G5IFVVZQFLXYQ, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.022216365 (624h48m16.8932994s) [noise=true , address=145.79.227.24:4614 , piece_id=6BJGK6OQWTHZCY3GQSQV7LVJWJOAJSFQSSVPDJ3RGX4EUFIISFRA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.036007444 (624h48m16.907090479s) [noise=true , address=5.231.73.3:4030 , piece_id=II77J2BA2EEOJXS4KPGI3GTC4AYQ5Z6NL4ABJCGP2XZEFBP5ABOA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.142349752 (624h48m17.013432787s) [noise=true , address=77.90.31.13:4657 , piece_id=5ITVL3WNXVLJLNGPJS5YSLTOSLULDQGK6F64TJQ6K52MZG2UBIMQ, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.318258862 (624h48m17.189341897s) [noise=true , address=77.90.31.16:4760 , piece_id=P56LD3DX5YLE2MPV3RORGLX25G5YDT2TNOVCA5CLGGBYZN6LUUTQ, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.326042824 (624h48m17.197125859s) [noise=true , address=45.91.116.9:4271 , piece_id=VZJVLEOMS452TXPJOMIVF4K36YRPLQW3EGZNMEB4BZ7HK7PTDQXQ, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.704912834 (624h48m17.575995869s) [noise=true , address=45.91.116.8:4066 , piece_id=DJA5A7MOGJUHX6QKJ32WG3S6HUXE5PXD4DOOMCHZSVZ3FQCFUNRQ, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.808151081 (624h48m17.679234116s) [noise=true , address=145.79.227.35:4492 , piece_id=5KJRZLQ3H3S6SFZWLTKRQU3JIN7RY4E5FJFDDMPHCY6LCYXTVOPA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.941220429 (624h48m17.812303464s) [noise=true , address=145.79.227.44:4219 , piece_id=TQUYQDIUJWUX43HZCUOHECBX77YNCL2SZ6BMGX2VQ4T2XUPWNKJA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:18.110531386 (624h48m17.981614421s) [noise=true , address=145.79.227.33:4894 , piece_id=MYHRMEIWJDASEUHHTGRKAN5IRNE7QBTO4VNX62LHJVGBIJS5UVLA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:18.110557937 (624h48m17.981640982s) [seg_pos={0 0}, upload=547498]: Segment reached optimal threshold of 14 pieces.
Dec 31 16:17:18.110785891 (624h48m17.981868956s) [noise=true , address=45.91.116.10:4093 , piece_id=ILTD2BF7GDMGAP7YLYZPEIFHDNX5BJZ5E4KFOUTKEXWF7NNFI64A, seg_pos={0 0}, upload=547498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ): protocol: expected piece hash; context canceled; EOF
Dec 31 16:17:18.110795980 (624h48m17.981879035s) [noise=true , address=45.91.116.10:4093 , piece_id=ILTD2BF7GDMGAP7YLYZPEIFHDNX5BJZ5E4KFOUTKEXWF7NNFI64A, seg_pos={0 0}, upload=547498]: Long tail cancellation due to optimal threshold
Dec 31 16:17:18.110806109 (624h48m17.981889144s) [noise=true , address=77.90.31.12:4606 , piece_id=CMC465DGAG3VRLPTRWL2TNC6MD3HFPIABJJQUTHVSL32LFWPQRBA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pLKmTja7ttT6vqjq8vHMYR9RLfnpqxY9xm9GDBG8ez9aHujs): protocol: write tcp 188.166.225.170:48024->77.90.31.12:4606: use of closed network connection; write tcp 188.166.225.170:48024->77.90.31.12:4606: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48024->77.90.31.12:4606: use of closed network connection
Dec 31 16:17:18.110813563 (624h48m17.981896608s) [noise=true , address=77.90.31.12:4606 , piece_id=CMC465DGAG3VRLPTRWL2TNC6MD3HFPIABJJQUTHVSL32LFWPQRBA, seg_pos={0 0}, upload=547498]: Long tail cancellation due to optimal threshold
Dec 31 16:17:18.111107804 (624h48m17.982190879s) [noise=true , address=45.91.116.7:4027 , piece_id=3AESXZLKXV7X2HH3OJHSDPMDY2OHI2GAFIAAFYJMOVFAJDU2FAUA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF
Dec 31 16:17:18.111117783 (624h48m17.982200828s) [noise=true , address=45.91.116.7:4027 , piece_id=3AESXZLKXV7X2HH3OJHSDPMDY2OHI2GAFIAAFYJMOVFAJDU2FAUA, seg_pos={0 0}, upload=547498]: Long tail cancellation due to optimal threshold
Dec 31 16:17:18.111224586 (624h48m17.982307671s) [noise=true , address=77.90.31.17:4788 , piece_id=4G5Z3ECGM5HBH7PELBGGDKXGDPHTNRHKEBXFEKVEP5ZDFXVIGWUA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: read tcp 188.166.225.170:50208->77.90.31.17:4788: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 16:17:18.111233905 (624h48m17.98231694s) [noise=true , address=77.90.31.17:4788 , piece_id=4G5Z3ECGM5HBH7PELBGGDKXGDPHTNRHKEBXFEKVEP5ZDFXVIGWUA, seg_pos={0 0}, upload=547498]: Long tail cancellation due to optimal threshold
Dec 31 16:17:18.111271807 (624h48m17.982354892s) [seg_pos={0 0}, upload=547498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 16:17:18.111364793 (624h48m17.982447838s) [upload=547498]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:17:18.111372739 (624h48m17.982455784s) [upload=547498]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:17:18.397617841 (624h48m18.268700876s) [upload=547498]: Done uploading segments...
Dec 31 16:17:18.397637810 (624h48m18.268720845s) [upload=547498]: Upload finished. err: <nil>
Duration: 48.511784571s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
5.987646847s trace:
https://jaeger.dev.storj.io/trace/5582e72abe7a8819
FAIL (2025-12-31 16:50:53 UTC)
orbiter warning: context deadline exceeded
Duration: 5.987675401s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
34 out of 60 is failed (threshold=6) [X..X...X....XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX] duration:
5.001138139s trace:
https://jaeger.dev.storj.io/trace/19c6aa42963b3c0f
FAIL (2025-12-31 16:50:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001162844s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [X...XX..XX........X....X.X...X....X.X.......X.XX..X..X...X.X] duration:
30.002973747s trace:
https://jaeger.dev.storj.io/trace/2bd44400444d3a18
FAIL (2025-12-31 16:50:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: WPT5LIUIRYEBO2ZDDGNLPJ3NBTAR7OSOBI3SUG7QLVUDJ33NU55A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: MBK7ZDCWDBFVBPK7BK7YYMLRC3KJZBGE3TBXR2TYCBLAVFDHYJTQ) context deadline exceeded
======LOGS======
Dec 31 16:50:06.195526122 (625h21m6.066609167s) [upload=547974]: Starting upload
Dec 31 16:50:06.195619359 (625h21m6.066702394s) [upload=547974]: Done starting upload
Dec 31 16:50:06.195695003 (625h21m6.066778048s) [upload=547974]: Uploading segments...
Dec 31 16:50:06.196071881 (625h21m6.067154926s) [upload=547974]: Got next segment. Inline: false
Dec 31 16:50:06.196113159 (625h21m6.067196205s) [upload=547974]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 16:50:06.196118990 (625h21m6.067202035s) [upload=547974]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 16:50:06.443263899 (625h21m6.314346944s) [seg_pos={0 0}, upload=547974]: Begin upload segment...
Dec 31 16:50:06.443330886 (625h21m6.314413931s) [seg_pos={0 0}, upload=547974]: Done begin upload segment.
Dec 31 16:50:06.443374961 (625h21m6.314458016s) [noise=true , address=145.79.227.6:4257 , piece_id=T5FDGD74QDOUXLHJ4Q2EHOM5U3TLWBTFMY2U4B7FHW53RJNG6H3A, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.443405529 (625h21m6.314488564s) [noise=true , address=145.79.227.9:4367 , piece_id=APZKQVIBW66G4FHIJPK2IGQZRLHMG7NKKK336HPHH6ZVK4454SMQ, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.444110832 (625h21m6.315193877s) [noise=true , address=77.90.31.12:4589 , piece_id=WPT5LIUIRYEBO2ZDDGNLPJ3NBTAR7OSOBI3SUG7QLVUDJ33NU55A, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.444640640 (625h21m6.315723685s) [noise=true , address=45.91.116.11:4120 , piece_id=SXXI2AWRCTQA2L25E2VKQPGARWWONK25WCKILETCOJH7Y6JAVO2A, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.445877787 (625h21m6.316960832s) [noise=true , address=145.79.227.3:4150 , piece_id=IC3HP2IF4TIG6XG7TCGD6IJO264QIP46YIMPJOYHC7J3FPZJUMKA, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.446447671 (625h21m6.317530716s) [noise=true , address=145.79.227.21:4474 , piece_id=O62BG3LXFAZLZAM6KIYXVUC7I4WG3LCJUFX6S6JH4NWB24L2HHAQ, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.446917465 (625h21m6.31800051s) [noise=true , address=45.91.116.5:4300 , piece_id=MBK7ZDCWDBFVBPK7BK7YYMLRC3KJZBGE3TBXR2TYCBLAVFDHYJTQ, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.447282882 (625h21m6.318365927s) [noise=true , address=145.79.227.14:4839 , piece_id=NTG7XLUKRUHRV5GZSOLCZIHFFDUKLELOGQTSAQ6VX5P5JU4Z4WLA, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.447618731 (625h21m6.318701766s) [noise=true , address=145.79.227.29:4941 , piece_id=WSHSFOSXDXPRYR2PDPSXMPCP5VYCIM4GRIGWGVMKZKYDCIJVMGZQ, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.448009495 (625h21m6.319092541s) [noise=true , address=145.79.227.20:4437 , piece_id=Y5HPFVGLYUV7NZKX6N56GAIBE6HO3R4WF6WK27FR2HBZSK7KYOVQ, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.448452558 (625h21m6.319535603s) [noise=true , address=145.79.227.51:4410 , piece_id=J6V53T5NJKA4WR37XP4QHEOM3RPAKNG52YFMR5KEXWYTY2XDKZZA, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.448824156 (625h21m6.319907191s) [noise=true , address=145.79.227.50:4373 , piece_id=E3KLYBUJAMKV4S2UYRCVJ46YGD2VFR6WH2ERFVD6WTN2ZBC67NJA, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.449305343 (625h21m6.320388388s) [noise=true , address=145.79.227.45:4231 , piece_id=VDZQMBUR6WHXWABVHPLTVC5IJMPQYDW2X2XIDJCCHEZEIUGPDXPQ, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.449760189 (625h21m6.320843234s) [noise=true , address=77.90.31.13:4627 , piece_id=JC7S4L3N5RQG7MDCKGHF63UX6YNNWRNQN4EGMMB3XFGAXGEMBJQQ, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.450367615 (625h21m6.32145066s) [noise=true , address=45.91.116.6:4973 , piece_id=KOXRKS25AF6P2IYUHSJ5OEOQWZA3LNGDXM4EUSLPVJY57IESHRCQ, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:06.450818033 (625h21m6.321901068s) [noise=true , address=145.79.227.15:4876 , piece_id=6XVXA3CTQMTDUVSU6U2ZZPEH3Y6AQXDYKZMBHR6Q5NUYHYNL677Q, seg_pos={0 0}, upload=547974]: Uploading piece...
Dec 31 16:50:08.114790637 (625h21m7.985873672s) [upload=547974]: Next returned nil segment
Dec 31 16:50:08.114805166 (625h21m7.985888211s) [upload=547974]: Waiting for error group managing segments...
Dec 31 16:50:08.425962442 (625h21m8.297045477s) [noise=true , address=145.79.227.51:4410 , piece_id=J6V53T5NJKA4WR37XP4QHEOM3RPAKNG52YFMR5KEXWYTY2XDKZZA, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.427604178 (625h21m8.298687223s) [noise=true , address=145.79.227.9:4367 , piece_id=APZKQVIBW66G4FHIJPK2IGQZRLHMG7NKKK336HPHH6ZVK4454SMQ, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.435654106 (625h21m8.306737141s) [noise=true , address=145.79.227.3:4150 , piece_id=IC3HP2IF4TIG6XG7TCGD6IJO264QIP46YIMPJOYHC7J3FPZJUMKA, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.439789290 (625h21m8.310872325s) [noise=true , address=145.79.227.50:4373 , piece_id=E3KLYBUJAMKV4S2UYRCVJ46YGD2VFR6WH2ERFVD6WTN2ZBC67NJA, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.439893809 (625h21m8.310976854s) [noise=true , address=145.79.227.45:4231 , piece_id=VDZQMBUR6WHXWABVHPLTVC5IJMPQYDW2X2XIDJCCHEZEIUGPDXPQ, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.445041882 (625h21m8.316124957s) [noise=true , address=145.79.227.15:4876 , piece_id=6XVXA3CTQMTDUVSU6U2ZZPEH3Y6AQXDYKZMBHR6Q5NUYHYNL677Q, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.450778004 (625h21m8.32186105s) [noise=true , address=145.79.227.6:4257 , piece_id=T5FDGD74QDOUXLHJ4Q2EHOM5U3TLWBTFMY2U4B7FHW53RJNG6H3A, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.452441742 (625h21m8.323524787s) [noise=true , address=145.79.227.20:4437 , piece_id=Y5HPFVGLYUV7NZKX6N56GAIBE6HO3R4WF6WK27FR2HBZSK7KYOVQ, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.459060124 (625h21m8.330143179s) [noise=true , address=145.79.227.21:4474 , piece_id=O62BG3LXFAZLZAM6KIYXVUC7I4WG3LCJUFX6S6JH4NWB24L2HHAQ, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.493893091 (625h21m8.364976137s) [noise=true , address=145.79.227.14:4839 , piece_id=NTG7XLUKRUHRV5GZSOLCZIHFFDUKLELOGQTSAQ6VX5P5JU4Z4WLA, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.630734005 (625h21m8.50181705s) [noise=true , address=77.90.31.12:4589 , piece_id=WPT5LIUIRYEBO2ZDDGNLPJ3NBTAR7OSOBI3SUG7QLVUDJ33NU55A, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.792135947 (625h21m8.663218992s) [noise=true , address=45.91.116.6:4973 , piece_id=KOXRKS25AF6P2IYUHSJ5OEOQWZA3LNGDXM4EUSLPVJY57IESHRCQ, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.919533560 (625h21m8.790616595s) [noise=true , address=45.91.116.5:4300 , piece_id=MBK7ZDCWDBFVBPK7BK7YYMLRC3KJZBGE3TBXR2TYCBLAVFDHYJTQ, seg_pos={0 0}, upload=547974]: Done uploading piece. err: <nil>
Dec 31 16:50:08.919556524 (625h21m8.790639569s) [seg_pos={0 0}, upload=547974]: Segment reached optimal threshold of 13 pieces.
Dec 31 16:50:08.919771343 (625h21m8.790854388s) [noise=true , address=145.79.227.29:4941 , piece_id=WSHSFOSXDXPRYR2PDPSXMPCP5VYCIM4GRIGWGVMKZKYDCIJVMGZQ, seg_pos={0 0}, upload=547974]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 31 16:50:08.919778897 (625h21m8.790861942s) [noise=true , address=145.79.227.29:4941 , piece_id=WSHSFOSXDXPRYR2PDPSXMPCP5VYCIM4GRIGWGVMKZKYDCIJVMGZQ, seg_pos={0 0}, upload=547974]: Long tail cancellation due to optimal threshold
Dec 31 16:50:08.920124055 (625h21m8.7912071s) [noise=true , address=77.90.31.13:4627 , piece_id=JC7S4L3N5RQG7MDCKGHF63UX6YNNWRNQN4EGMMB3XFGAXGEMBJQQ, seg_pos={0 0}, upload=547974]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: read tcp 188.166.225.170:54978->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 16:50:08.920131669 (625h21m8.791214704s) [noise=true , address=77.90.31.13:4627 , piece_id=JC7S4L3N5RQG7MDCKGHF63UX6YNNWRNQN4EGMMB3XFGAXGEMBJQQ, seg_pos={0 0}, upload=547974]: Long tail cancellation due to optimal threshold
Dec 31 16:50:08.920151146 (625h21m8.791234202s) [noise=true , address=45.91.116.11:4120 , piece_id=SXXI2AWRCTQA2L25E2VKQPGARWWONK25WCKILETCOJH7Y6JAVO2A, seg_pos={0 0}, upload=547974]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: read tcp 188.166.225.170:49932->45.91.116.11:4120: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 16:50:08.920171414 (625h21m8.791254449s) [noise=true , address=45.91.116.11:4120 , piece_id=SXXI2AWRCTQA2L25E2VKQPGARWWONK25WCKILETCOJH7Y6JAVO2A, seg_pos={0 0}, upload=547974]: Long tail cancellation due to optimal threshold
Dec 31 16:50:08.920200039 (625h21m8.791283084s) [seg_pos={0 0}, upload=547974]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 16:50:08.920314899 (625h21m8.791397944s) [upload=547974]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:50:08.920322423 (625h21m8.791405468s) [upload=547974]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:50:09.213054485 (625h21m9.08413752s) [upload=547974]: Done uploading segments...
Dec 31 16:50:09.213076607 (625h21m9.084159652s) [upload=547974]: Upload finished. err: <nil>
Duration: 30.002997613s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
12.342419571s trace:
https://jaeger.dev.storj.io/trace/5582e72abe7a8819
FAIL (2025-12-31 16:50:20 UTC)
orbiter warning: context deadline exceeded
Duration: 12.342438666s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [.X..X...X....XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XX] duration:
5.034351213s trace:
https://jaeger.dev.storj.io/trace/1b62a04c8c462de
FAIL (2025-12-31 16:50:16 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.034412219s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [............X..........X....................................] duration:
36.921533682s trace:
https://jaeger.dev.storj.io/trace/5b4dfa890c7fc19f
FAIL (2025-12-31 16:49:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
QCFUNRQ, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.048997877 (624h48m14.920080912s) [noise=true , address=77.90.31.13:4657 , piece_id=5ITVL3WNXVLJLNGPJS5YSLTOSLULDQGK6F64TJQ6K52MZG2UBIMQ, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.049460398 (624h48m14.920543463s) [noise=true , address=145.79.227.6:4278 , piece_id=UPW54EA7CL2QDHHSBOCOUHOJ4PO664J7HPFRMKDZVRJWLQLZQE3Q, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.049958216 (624h48m14.921041251s) [noise=true , address=77.90.31.12:4606 , piece_id=CMC465DGAG3VRLPTRWL2TNC6MD3HFPIABJJQUTHVSL32LFWPQRBA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.049996479 (624h48m14.921079514s) [noise=true , address=145.79.227.35:4492 , piece_id=5KJRZLQ3H3S6SFZWLTKRQU3JIN7RY4E5FJFDDMPHCY6LCYXTVOPA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.050251765 (624h48m14.9213348s) [noise=true , address=5.231.73.3:4030 , piece_id=II77J2BA2EEOJXS4KPGI3GTC4AYQ5Z6NL4ABJCGP2XZEFBP5ABOA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.050429022 (624h48m14.921512057s) [noise=true , address=45.91.116.9:4271 , piece_id=VZJVLEOMS452TXPJOMIVF4K36YRPLQW3EGZNMEB4BZ7HK7PTDQXQ, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.050521198 (624h48m14.921604233s) [noise=true , address=145.79.227.24:4614 , piece_id=6BJGK6OQWTHZCY3GQSQV7LVJWJOAJSFQSSVPDJ3RGX4EUFIISFRA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.050715528 (624h48m14.921798573s) [noise=true , address=145.79.227.23:4575 , piece_id=NXYLJMIMKJYMJ2UF5BYXYODDUOCROHYNDEHXCO2G5IFVVZQFLXYQ, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.050816169 (624h48m14.921899204s) [noise=true , address=145.79.227.38:4004 , piece_id=OMFDOFGQ3EBWZPON3I3DP6LWBP3BWWEEFHU34EXZBF2BAJLP3TNQ, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.050998136 (624h48m14.922081171s) [noise=true , address=145.79.227.17:4985 , piece_id=Q37B2D24JXNSSJIEIBRSYEE6IEOQV7IG4SEIGLFWLW2PBTWZU5TA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.051085192 (624h48m14.922168227s) [noise=true , address=145.79.227.44:4219 , piece_id=TQUYQDIUJWUX43HZCUOHECBX77YNCL2SZ6BMGX2VQ4T2XUPWNKJA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.051462040 (624h48m14.922545075s) [noise=true , address=77.90.31.17:4788 , piece_id=4G5Z3ECGM5HBH7PELBGGDKXGDPHTNRHKEBXFEKVEP5ZDFXVIGWUA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.051871809 (624h48m14.922954844s) [noise=true , address=77.90.31.16:4760 , piece_id=P56LD3DX5YLE2MPV3RORGLX25G5YDT2TNOVCA5CLGGBYZN6LUUTQ, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.052116004 (624h48m14.923199039s) [noise=true , address=145.79.227.33:4894 , piece_id=MYHRMEIWJDASEUHHTGRKAN5IRNE7QBTO4VNX62LHJVGBIJS5UVLA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.052414814 (624h48m14.923497859s) [noise=true , address=145.79.227.19:4087 , piece_id=7EXAHR36NSDTHNUWQQZTIS46XWFQUWTKAEIPTMSQHZHDJHCKI6PA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:16.513009118 (624h48m16.384092153s) [upload=547498]: Next returned nil segment
Dec 31 16:17:16.513023495 (624h48m16.38410654s) [upload=547498]: Waiting for error group managing segments...
Dec 31 16:17:16.908124481 (624h48m16.779207526s) [noise=true , address=145.79.227.19:4087 , piece_id=7EXAHR36NSDTHNUWQQZTIS46XWFQUWTKAEIPTMSQHZHDJHCKI6PA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:16.918737279 (624h48m16.789820325s) [noise=true , address=145.79.227.6:4278 , piece_id=UPW54EA7CL2QDHHSBOCOUHOJ4PO664J7HPFRMKDZVRJWLQLZQE3Q, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:16.964651124 (624h48m16.835734159s) [noise=true , address=145.79.227.17:4985 , piece_id=Q37B2D24JXNSSJIEIBRSYEE6IEOQV7IG4SEIGLFWLW2PBTWZU5TA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:16.994270222 (624h48m16.865353258s) [noise=true , address=145.79.227.38:4004 , piece_id=OMFDOFGQ3EBWZPON3I3DP6LWBP3BWWEEFHU34EXZBF2BAJLP3TNQ, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:16.997634448 (624h48m16.868717483s) [noise=true , address=145.79.227.23:4575 , piece_id=NXYLJMIMKJYMJ2UF5BYXYODDUOCROHYNDEHXCO2G5IFVVZQFLXYQ, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.022216365 (624h48m16.8932994s) [noise=true , address=145.79.227.24:4614 , piece_id=6BJGK6OQWTHZCY3GQSQV7LVJWJOAJSFQSSVPDJ3RGX4EUFIISFRA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.036007444 (624h48m16.907090479s) [noise=true , address=5.231.73.3:4030 , piece_id=II77J2BA2EEOJXS4KPGI3GTC4AYQ5Z6NL4ABJCGP2XZEFBP5ABOA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.142349752 (624h48m17.013432787s) [noise=true , address=77.90.31.13:4657 , piece_id=5ITVL3WNXVLJLNGPJS5YSLTOSLULDQGK6F64TJQ6K52MZG2UBIMQ, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.318258862 (624h48m17.189341897s) [noise=true , address=77.90.31.16:4760 , piece_id=P56LD3DX5YLE2MPV3RORGLX25G5YDT2TNOVCA5CLGGBYZN6LUUTQ, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.326042824 (624h48m17.197125859s) [noise=true , address=45.91.116.9:4271 , piece_id=VZJVLEOMS452TXPJOMIVF4K36YRPLQW3EGZNMEB4BZ7HK7PTDQXQ, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.704912834 (624h48m17.575995869s) [noise=true , address=45.91.116.8:4066 , piece_id=DJA5A7MOGJUHX6QKJ32WG3S6HUXE5PXD4DOOMCHZSVZ3FQCFUNRQ, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.808151081 (624h48m17.679234116s) [noise=true , address=145.79.227.35:4492 , piece_id=5KJRZLQ3H3S6SFZWLTKRQU3JIN7RY4E5FJFDDMPHCY6LCYXTVOPA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.941220429 (624h48m17.812303464s) [noise=true , address=145.79.227.44:4219 , piece_id=TQUYQDIUJWUX43HZCUOHECBX77YNCL2SZ6BMGX2VQ4T2XUPWNKJA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:18.110531386 (624h48m17.981614421s) [noise=true , address=145.79.227.33:4894 , piece_id=MYHRMEIWJDASEUHHTGRKAN5IRNE7QBTO4VNX62LHJVGBIJS5UVLA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:18.110557937 (624h48m17.981640982s) [seg_pos={0 0}, upload=547498]: Segment reached optimal threshold of 14 pieces.
Dec 31 16:17:18.110785891 (624h48m17.981868956s) [noise=true , address=45.91.116.10:4093 , piece_id=ILTD2BF7GDMGAP7YLYZPEIFHDNX5BJZ5E4KFOUTKEXWF7NNFI64A, seg_pos={0 0}, upload=547498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ): protocol: expected piece hash; context canceled; EOF
Dec 31 16:17:18.110795980 (624h48m17.981879035s) [noise=true , address=45.91.116.10:4093 , piece_id=ILTD2BF7GDMGAP7YLYZPEIFHDNX5BJZ5E4KFOUTKEXWF7NNFI64A, seg_pos={0 0}, upload=547498]: Long tail cancellation due to optimal threshold
Dec 31 16:17:18.110806109 (624h48m17.981889144s) [noise=true , address=77.90.31.12:4606 , piece_id=CMC465DGAG3VRLPTRWL2TNC6MD3HFPIABJJQUTHVSL32LFWPQRBA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pLKmTja7ttT6vqjq8vHMYR9RLfnpqxY9xm9GDBG8ez9aHujs): protocol: write tcp 188.166.225.170:48024->77.90.31.12:4606: use of closed network connection; write tcp 188.166.225.170:48024->77.90.31.12:4606: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48024->77.90.31.12:4606: use of closed network connection
Dec 31 16:17:18.110813563 (624h48m17.981896608s) [noise=true , address=77.90.31.12:4606 , piece_id=CMC465DGAG3VRLPTRWL2TNC6MD3HFPIABJJQUTHVSL32LFWPQRBA, seg_pos={0 0}, upload=547498]: Long tail cancellation due to optimal threshold
Dec 31 16:17:18.111107804 (624h48m17.982190879s) [noise=true , address=45.91.116.7:4027 , piece_id=3AESXZLKXV7X2HH3OJHSDPMDY2OHI2GAFIAAFYJMOVFAJDU2FAUA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF
Dec 31 16:17:18.111117783 (624h48m17.982200828s) [noise=true , address=45.91.116.7:4027 , piece_id=3AESXZLKXV7X2HH3OJHSDPMDY2OHI2GAFIAAFYJMOVFAJDU2FAUA, seg_pos={0 0}, upload=547498]: Long tail cancellation due to optimal threshold
Dec 31 16:17:18.111224586 (624h48m17.982307671s) [noise=true , address=77.90.31.17:4788 , piece_id=4G5Z3ECGM5HBH7PELBGGDKXGDPHTNRHKEBXFEKVEP5ZDFXVIGWUA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: read tcp 188.166.225.170:50208->77.90.31.17:4788: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 16:17:18.111233905 (624h48m17.98231694s) [noise=true , address=77.90.31.17:4788 , piece_id=4G5Z3ECGM5HBH7PELBGGDKXGDPHTNRHKEBXFEKVEP5ZDFXVIGWUA, seg_pos={0 0}, upload=547498]: Long tail cancellation due to optimal threshold
Dec 31 16:17:18.111271807 (624h48m17.982354892s) [seg_pos={0 0}, upload=547498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 16:17:18.111364793 (624h48m17.982447838s) [upload=547498]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:17:18.111372739 (624h48m17.982455784s) [upload=547498]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:17:18.397617841 (624h48m18.268700876s) [upload=547498]: Done uploading segments...
Dec 31 16:17:18.397637810 (624h48m18.268720845s) [upload=547498]: Upload finished. err: <nil>
Duration: 36.921549332s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
32 out of 60 is failed (threshold=6) [..X..X...X....XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.X] duration:
5.001656557s trace:
https://jaeger.dev.storj.io/trace/2ea81d7f170cc0d4
FAIL (2025-12-31 16:49:51 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001683388s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [.X...XX..XX........X....X.X...X....X.X.......X.XX..X..X...X.] duration:
6.065987733s trace:
https://jaeger.dev.storj.io/trace/360f4afa7a93c349
FAIL (2025-12-31 16:49:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
c 31 16:48:57.540897764 (625h19m57.411980809s) [upload=547957]: Starting upload
Dec 31 16:48:57.540962337 (625h19m57.412045372s) [upload=547957]: Done starting upload
Dec 31 16:48:57.541156758 (625h19m57.412239793s) [upload=547957]: Uploading segments...
Dec 31 16:48:57.541212012 (625h19m57.412295057s) [upload=547957]: Got next segment. Inline: false
Dec 31 16:48:57.541239856 (625h19m57.412322891s) [upload=547957]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 16:48:57.541244214 (625h19m57.412327259s) [upload=547957]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 16:48:57.786764999 (625h19m57.657848034s) [seg_pos={0 0}, upload=547957]: Begin upload segment...
Dec 31 16:48:57.786829522 (625h19m57.657912567s) [seg_pos={0 0}, upload=547957]: Done begin upload segment.
Dec 31 16:48:57.786840764 (625h19m57.657923799s) [noise=true , address=145.79.227.4:4187 , piece_id=6EMGX2JYVOWQZRZYU6BMRZQ2S4OKEICE4DT3RAOAQCZRNLBNA3BA, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.786866051 (625h19m57.657949086s) [noise=true , address=77.90.31.11:4555 , piece_id=UY3CVJMQIMTTSEXNZAW64XI5QHDEY5F35YUBUTGQ72JM2J77TTZA, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.787386803 (625h19m57.658469848s) [noise=true , address=45.91.116.14:4191 , piece_id=WFJHMTQ2MJLTIKSNP27UVDCYD4R3L4NUB63QYYNEH4NZVFYSCOAA, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.787840086 (625h19m57.658923121s) [noise=true , address=145.79.227.47:4266 , piece_id=NXTNDOVZOHXRUGC6VG6DYIBTMEZNPENTLQZ2MYKQVA64KSLY7MHA, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.788252672 (625h19m57.659335707s) [noise=true , address=45.91.116.10:4082 , piece_id=G6WZTM5KTM5QMFDLDPGNCNGAJIOKTQT7KCZGT7GS5DIYXRGDGV5A, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.788673453 (625h19m57.659756488s) [noise=true , address=145.79.227.10:4401 , piece_id=ZZYQROWFOLL5ZRDZULTWDDRQOW5C6KRLRKGONVZEKBGGUJYDWKWQ, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.789333029 (625h19m57.660416064s) [noise=true , address=77.90.31.10:4519 , piece_id=FT2CNULX4ZHKCMMRAE5F22QP6MPM26TQ63GDAGBN5ZGXIZBBSY2A, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.789740305 (625h19m57.66082334s) [noise=true , address=145.79.227.20:4439 , piece_id=GWOLM7Y4RHJFASHE65XZXQMLDGRG7J7RFX5LMD7UYPES53Z3KMNQ, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.791228248 (625h19m57.662311293s) [noise=true , address=145.79.227.24:4581 , piece_id=ZMFNMVZQWWGRU6OYMZFTEYG57E6OA6BMVYSU445LIXMXS3KHNSMQ, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.792491714 (625h19m57.663574749s) [noise=true , address=45.91.116.6:4974 , piece_id=FNNIPF42VMKR4SN6LRLYXVGZF66IBSSUOUU3VB5GWBEBUHUFDHFQ, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.793733459 (625h19m57.664816504s) [noise=true , address=145.79.227.50:4375 , piece_id=DJLFB5RWI6ZZBTL2CLHPOK4EXGCD5VKP5UBE752XTBNWD7ULTLWA, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.794783919 (625h19m57.665866954s) [noise=true , address=145.79.227.6:4259 , piece_id=TJ6VEWUI2BHVRBUED5354TSZHS5P37HMEQGM647JAMO4KCRL74CA, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.795852904 (625h19m57.666935949s) [noise=true , address=145.79.227.22:4509 , piece_id=FP75JCGGNGIVH4M7GZ62ALXN4VMM2OFG64W7XPP2XNAK4DOJ3YZA, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.796440322 (625h19m57.667523367s) [noise=true , address=145.79.227.7:4293 , piece_id=QCWLWRTQUTJXDNTDPZEBDQ2LCRQAFNQWSFZOLVGEFR5LBIM4VUWQ, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.797835409 (625h19m57.668918444s) [noise=true , address=77.90.31.15:4700 , piece_id=JIAERY6LJQ2KM7RC2SABJYENPG54YKE4K7UTTN7ZKYBHD45BR5UA, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.797951730 (625h19m57.669034775s) [noise=true , address=145.79.227.51:4409 , piece_id=DEI6FIKTHDK3SEUWMXUVHG6DULGD3N5O66A4IHMK2DPR3UFZ4AHQ, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:59.486904330 (625h19m59.357987365s) [upload=547957]: Next returned nil segment
Dec 31 16:48:59.486926953 (625h19m59.358009988s) [upload=547957]: Waiting for error group managing segments...
Dec 31 16:48:59.766821791 (625h19m59.637904836s) [noise=true , address=145.79.227.7:4293 , piece_id=QCWLWRTQUTJXDNTDPZEBDQ2LCRQAFNQWSFZOLVGEFR5LBIM4VUWQ, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.779066197 (625h19m59.650149242s) [noise=true , address=145.79.227.6:4259 , piece_id=TJ6VEWUI2BHVRBUED5354TSZHS5P37HMEQGM647JAMO4KCRL74CA, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.781388027 (625h19m59.652471072s) [noise=true , address=145.79.227.4:4187 , piece_id=6EMGX2JYVOWQZRZYU6BMRZQ2S4OKEICE4DT3RAOAQCZRNLBNA3BA, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.785669350 (625h19m59.656752395s) [noise=true , address=145.79.227.20:4439 , piece_id=GWOLM7Y4RHJFASHE65XZXQMLDGRG7J7RFX5LMD7UYPES53Z3KMNQ, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.786080062 (625h19m59.657163097s) [noise=true , address=145.79.227.22:4509 , piece_id=FP75JCGGNGIVH4M7GZ62ALXN4VMM2OFG64W7XPP2XNAK4DOJ3YZA, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.789371500 (625h19m59.660454535s) [noise=true , address=145.79.227.10:4401 , piece_id=ZZYQROWFOLL5ZRDZULTWDDRQOW5C6KRLRKGONVZEKBGGUJYDWKWQ, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.797668599 (625h19m59.668751644s) [noise=true , address=145.79.227.50:4375 , piece_id=DJLFB5RWI6ZZBTL2CLHPOK4EXGCD5VKP5UBE752XTBNWD7ULTLWA, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.802629895 (625h19m59.67371294s) [noise=true , address=145.79.227.51:4409 , piece_id=DEI6FIKTHDK3SEUWMXUVHG6DULGD3N5O66A4IHMK2DPR3UFZ4AHQ, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.805777438 (625h19m59.676860473s) [noise=true , address=145.79.227.47:4266 , piece_id=NXTNDOVZOHXRUGC6VG6DYIBTMEZNPENTLQZ2MYKQVA64KSLY7MHA, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.867068175 (625h19m59.73815122s) [noise=true , address=145.79.227.24:4581 , piece_id=ZMFNMVZQWWGRU6OYMZFTEYG57E6OA6BMVYSU445LIXMXS3KHNSMQ, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.952097772 (625h19m59.823180837s) [noise=true , address=77.90.31.11:4555 , piece_id=UY3CVJMQIMTTSEXNZAW64XI5QHDEY5F35YUBUTGQ72JM2J77TTZA, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.985181299 (625h19m59.856264334s) [noise=true , address=77.90.31.10:4519 , piece_id=FT2CNULX4ZHKCMMRAE5F22QP6MPM26TQ63GDAGBN5ZGXIZBBSY2A, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:49:00.008306191 (625h19m59.879389226s) [noise=true , address=77.90.31.15:4700 , piece_id=JIAERY6LJQ2KM7RC2SABJYENPG54YKE4K7UTTN7ZKYBHD45BR5UA, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:49:00.008330568 (625h19m59.879413613s) [seg_pos={0 0}, upload=547957]: Segment reached optimal threshold of 13 pieces.
Dec 31 16:49:00.008586114 (625h19m59.879669149s) [noise=true , address=45.91.116.14:4191 , piece_id=WFJHMTQ2MJLTIKSNP27UVDCYD4R3L4NUB63QYYNEH4NZVFYSCOAA, seg_pos={0 0}, upload=547957]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:40580->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:40580->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40580->45.91.116.14:4191: use of closed network connection
Dec 31 16:49:00.008596273 (625h19m59.879679328s) [noise=true , address=45.91.116.14:4191 , piece_id=WFJHMTQ2MJLTIKSNP27UVDCYD4R3L4NUB63QYYNEH4NZVFYSCOAA, seg_pos={0 0}, upload=547957]: Long tail cancellation due to optimal threshold
Dec 31 16:49:00.009124881 (625h19m59.880207926s) [noise=true , address=45.91.116.6:4974 , piece_id=FNNIPF42VMKR4SN6LRLYXVGZF66IBSSUOUU3VB5GWBEBUHUFDHFQ, seg_pos={0 0}, upload=547957]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Dec 31 16:49:00.009134679 (625h19m59.880217724s) [noise=true , address=45.91.116.6:4974 , piece_id=FNNIPF42VMKR4SN6LRLYXVGZF66IBSSUOUU3VB5GWBEBUHUFDHFQ, seg_pos={0 0}, upload=547957]: Long tail cancellation due to optimal threshold
Dec 31 16:49:00.009297650 (625h19m59.880380715s) [noise=true , address=45.91.116.10:4082 , piece_id=G6WZTM5KTM5QMFDLDPGNCNGAJIOKTQT7KCZGT7GS5DIYXRGDGV5A, seg_pos={0 0}, upload=547957]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF
Dec 31 16:49:00.009305915 (625h19m59.88038896s) [noise=true , address=45.91.116.10:4082 , piece_id=G6WZTM5KTM5QMFDLDPGNCNGAJIOKTQT7KCZGT7GS5DIYXRGDGV5A, seg_pos={0 0}, upload=547957]: Long tail cancellation due to optimal threshold
Dec 31 16:49:00.009326734 (625h19m59.880409779s) [seg_pos={0 0}, upload=547957]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 16:49:00.009409472 (625h19m59.880492507s) [upload=547957]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:49:00.009415443 (625h19m59.880498489s) [upload=547957]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:49:00.303501945 (625h20m0.17458499s) [upload=547957]: Done uploading segments...
Dec 31 16:49:00.303528255 (625h20m0.17461129s) [upload=547957]: Upload finished. err: <nil>
Duration: 6.066005316s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [..X...XX..XX........X....X.X...X....X.X.......X.XX..X..X...X] duration:
30.003477098s trace:
https://jaeger.dev.storj.io/trace/360f4afa7a93c349
FAIL (2025-12-31 16:49:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
c 31 16:48:57.540897764 (625h19m57.411980809s) [upload=547957]: Starting upload
Dec 31 16:48:57.540962337 (625h19m57.412045372s) [upload=547957]: Done starting upload
Dec 31 16:48:57.541156758 (625h19m57.412239793s) [upload=547957]: Uploading segments...
Dec 31 16:48:57.541212012 (625h19m57.412295057s) [upload=547957]: Got next segment. Inline: false
Dec 31 16:48:57.541239856 (625h19m57.412322891s) [upload=547957]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 16:48:57.541244214 (625h19m57.412327259s) [upload=547957]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 16:48:57.786764999 (625h19m57.657848034s) [seg_pos={0 0}, upload=547957]: Begin upload segment...
Dec 31 16:48:57.786829522 (625h19m57.657912567s) [seg_pos={0 0}, upload=547957]: Done begin upload segment.
Dec 31 16:48:57.786840764 (625h19m57.657923799s) [noise=true , address=145.79.227.4:4187 , piece_id=6EMGX2JYVOWQZRZYU6BMRZQ2S4OKEICE4DT3RAOAQCZRNLBNA3BA, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.786866051 (625h19m57.657949086s) [noise=true , address=77.90.31.11:4555 , piece_id=UY3CVJMQIMTTSEXNZAW64XI5QHDEY5F35YUBUTGQ72JM2J77TTZA, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.787386803 (625h19m57.658469848s) [noise=true , address=45.91.116.14:4191 , piece_id=WFJHMTQ2MJLTIKSNP27UVDCYD4R3L4NUB63QYYNEH4NZVFYSCOAA, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.787840086 (625h19m57.658923121s) [noise=true , address=145.79.227.47:4266 , piece_id=NXTNDOVZOHXRUGC6VG6DYIBTMEZNPENTLQZ2MYKQVA64KSLY7MHA, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.788252672 (625h19m57.659335707s) [noise=true , address=45.91.116.10:4082 , piece_id=G6WZTM5KTM5QMFDLDPGNCNGAJIOKTQT7KCZGT7GS5DIYXRGDGV5A, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.788673453 (625h19m57.659756488s) [noise=true , address=145.79.227.10:4401 , piece_id=ZZYQROWFOLL5ZRDZULTWDDRQOW5C6KRLRKGONVZEKBGGUJYDWKWQ, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.789333029 (625h19m57.660416064s) [noise=true , address=77.90.31.10:4519 , piece_id=FT2CNULX4ZHKCMMRAE5F22QP6MPM26TQ63GDAGBN5ZGXIZBBSY2A, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.789740305 (625h19m57.66082334s) [noise=true , address=145.79.227.20:4439 , piece_id=GWOLM7Y4RHJFASHE65XZXQMLDGRG7J7RFX5LMD7UYPES53Z3KMNQ, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.791228248 (625h19m57.662311293s) [noise=true , address=145.79.227.24:4581 , piece_id=ZMFNMVZQWWGRU6OYMZFTEYG57E6OA6BMVYSU445LIXMXS3KHNSMQ, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.792491714 (625h19m57.663574749s) [noise=true , address=45.91.116.6:4974 , piece_id=FNNIPF42VMKR4SN6LRLYXVGZF66IBSSUOUU3VB5GWBEBUHUFDHFQ, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.793733459 (625h19m57.664816504s) [noise=true , address=145.79.227.50:4375 , piece_id=DJLFB5RWI6ZZBTL2CLHPOK4EXGCD5VKP5UBE752XTBNWD7ULTLWA, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.794783919 (625h19m57.665866954s) [noise=true , address=145.79.227.6:4259 , piece_id=TJ6VEWUI2BHVRBUED5354TSZHS5P37HMEQGM647JAMO4KCRL74CA, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.795852904 (625h19m57.666935949s) [noise=true , address=145.79.227.22:4509 , piece_id=FP75JCGGNGIVH4M7GZ62ALXN4VMM2OFG64W7XPP2XNAK4DOJ3YZA, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.796440322 (625h19m57.667523367s) [noise=true , address=145.79.227.7:4293 , piece_id=QCWLWRTQUTJXDNTDPZEBDQ2LCRQAFNQWSFZOLVGEFR5LBIM4VUWQ, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.797835409 (625h19m57.668918444s) [noise=true , address=77.90.31.15:4700 , piece_id=JIAERY6LJQ2KM7RC2SABJYENPG54YKE4K7UTTN7ZKYBHD45BR5UA, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:57.797951730 (625h19m57.669034775s) [noise=true , address=145.79.227.51:4409 , piece_id=DEI6FIKTHDK3SEUWMXUVHG6DULGD3N5O66A4IHMK2DPR3UFZ4AHQ, seg_pos={0 0}, upload=547957]: Uploading piece...
Dec 31 16:48:59.486904330 (625h19m59.357987365s) [upload=547957]: Next returned nil segment
Dec 31 16:48:59.486926953 (625h19m59.358009988s) [upload=547957]: Waiting for error group managing segments...
Dec 31 16:48:59.766821791 (625h19m59.637904836s) [noise=true , address=145.79.227.7:4293 , piece_id=QCWLWRTQUTJXDNTDPZEBDQ2LCRQAFNQWSFZOLVGEFR5LBIM4VUWQ, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.779066197 (625h19m59.650149242s) [noise=true , address=145.79.227.6:4259 , piece_id=TJ6VEWUI2BHVRBUED5354TSZHS5P37HMEQGM647JAMO4KCRL74CA, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.781388027 (625h19m59.652471072s) [noise=true , address=145.79.227.4:4187 , piece_id=6EMGX2JYVOWQZRZYU6BMRZQ2S4OKEICE4DT3RAOAQCZRNLBNA3BA, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.785669350 (625h19m59.656752395s) [noise=true , address=145.79.227.20:4439 , piece_id=GWOLM7Y4RHJFASHE65XZXQMLDGRG7J7RFX5LMD7UYPES53Z3KMNQ, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.786080062 (625h19m59.657163097s) [noise=true , address=145.79.227.22:4509 , piece_id=FP75JCGGNGIVH4M7GZ62ALXN4VMM2OFG64W7XPP2XNAK4DOJ3YZA, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.789371500 (625h19m59.660454535s) [noise=true , address=145.79.227.10:4401 , piece_id=ZZYQROWFOLL5ZRDZULTWDDRQOW5C6KRLRKGONVZEKBGGUJYDWKWQ, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.797668599 (625h19m59.668751644s) [noise=true , address=145.79.227.50:4375 , piece_id=DJLFB5RWI6ZZBTL2CLHPOK4EXGCD5VKP5UBE752XTBNWD7ULTLWA, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.802629895 (625h19m59.67371294s) [noise=true , address=145.79.227.51:4409 , piece_id=DEI6FIKTHDK3SEUWMXUVHG6DULGD3N5O66A4IHMK2DPR3UFZ4AHQ, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.805777438 (625h19m59.676860473s) [noise=true , address=145.79.227.47:4266 , piece_id=NXTNDOVZOHXRUGC6VG6DYIBTMEZNPENTLQZ2MYKQVA64KSLY7MHA, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.867068175 (625h19m59.73815122s) [noise=true , address=145.79.227.24:4581 , piece_id=ZMFNMVZQWWGRU6OYMZFTEYG57E6OA6BMVYSU445LIXMXS3KHNSMQ, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.952097772 (625h19m59.823180837s) [noise=true , address=77.90.31.11:4555 , piece_id=UY3CVJMQIMTTSEXNZAW64XI5QHDEY5F35YUBUTGQ72JM2J77TTZA, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:48:59.985181299 (625h19m59.856264334s) [noise=true , address=77.90.31.10:4519 , piece_id=FT2CNULX4ZHKCMMRAE5F22QP6MPM26TQ63GDAGBN5ZGXIZBBSY2A, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:49:00.008306191 (625h19m59.879389226s) [noise=true , address=77.90.31.15:4700 , piece_id=JIAERY6LJQ2KM7RC2SABJYENPG54YKE4K7UTTN7ZKYBHD45BR5UA, seg_pos={0 0}, upload=547957]: Done uploading piece. err: <nil>
Dec 31 16:49:00.008330568 (625h19m59.879413613s) [seg_pos={0 0}, upload=547957]: Segment reached optimal threshold of 13 pieces.
Dec 31 16:49:00.008586114 (625h19m59.879669149s) [noise=true , address=45.91.116.14:4191 , piece_id=WFJHMTQ2MJLTIKSNP27UVDCYD4R3L4NUB63QYYNEH4NZVFYSCOAA, seg_pos={0 0}, upload=547957]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:40580->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:40580->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40580->45.91.116.14:4191: use of closed network connection
Dec 31 16:49:00.008596273 (625h19m59.879679328s) [noise=true , address=45.91.116.14:4191 , piece_id=WFJHMTQ2MJLTIKSNP27UVDCYD4R3L4NUB63QYYNEH4NZVFYSCOAA, seg_pos={0 0}, upload=547957]: Long tail cancellation due to optimal threshold
Dec 31 16:49:00.009124881 (625h19m59.880207926s) [noise=true , address=45.91.116.6:4974 , piece_id=FNNIPF42VMKR4SN6LRLYXVGZF66IBSSUOUU3VB5GWBEBUHUFDHFQ, seg_pos={0 0}, upload=547957]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Dec 31 16:49:00.009134679 (625h19m59.880217724s) [noise=true , address=45.91.116.6:4974 , piece_id=FNNIPF42VMKR4SN6LRLYXVGZF66IBSSUOUU3VB5GWBEBUHUFDHFQ, seg_pos={0 0}, upload=547957]: Long tail cancellation due to optimal threshold
Dec 31 16:49:00.009297650 (625h19m59.880380715s) [noise=true , address=45.91.116.10:4082 , piece_id=G6WZTM5KTM5QMFDLDPGNCNGAJIOKTQT7KCZGT7GS5DIYXRGDGV5A, seg_pos={0 0}, upload=547957]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF
Dec 31 16:49:00.009305915 (625h19m59.88038896s) [noise=true , address=45.91.116.10:4082 , piece_id=G6WZTM5KTM5QMFDLDPGNCNGAJIOKTQT7KCZGT7GS5DIYXRGDGV5A, seg_pos={0 0}, upload=547957]: Long tail cancellation due to optimal threshold
Dec 31 16:49:00.009326734 (625h19m59.880409779s) [seg_pos={0 0}, upload=547957]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 16:49:00.009409472 (625h19m59.880492507s) [upload=547957]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:49:00.009415443 (625h19m59.880498489s) [upload=547957]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:49:00.303501945 (625h20m0.17458499s) [upload=547957]: Done uploading segments...
Dec 31 16:49:00.303528255 (625h20m0.17461129s) [upload=547957]: Upload finished. err: <nil>
Duration: 30.003498638s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............X..........X...................................] duration:
45.189416427s trace:
https://jaeger.dev.storj.io/trace/5b4dfa890c7fc19f
FAIL (2025-12-31 16:49:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
QCFUNRQ, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.048997877 (624h48m14.920080912s) [noise=true , address=77.90.31.13:4657 , piece_id=5ITVL3WNXVLJLNGPJS5YSLTOSLULDQGK6F64TJQ6K52MZG2UBIMQ, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.049460398 (624h48m14.920543463s) [noise=true , address=145.79.227.6:4278 , piece_id=UPW54EA7CL2QDHHSBOCOUHOJ4PO664J7HPFRMKDZVRJWLQLZQE3Q, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.049958216 (624h48m14.921041251s) [noise=true , address=77.90.31.12:4606 , piece_id=CMC465DGAG3VRLPTRWL2TNC6MD3HFPIABJJQUTHVSL32LFWPQRBA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.049996479 (624h48m14.921079514s) [noise=true , address=145.79.227.35:4492 , piece_id=5KJRZLQ3H3S6SFZWLTKRQU3JIN7RY4E5FJFDDMPHCY6LCYXTVOPA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.050251765 (624h48m14.9213348s) [noise=true , address=5.231.73.3:4030 , piece_id=II77J2BA2EEOJXS4KPGI3GTC4AYQ5Z6NL4ABJCGP2XZEFBP5ABOA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.050429022 (624h48m14.921512057s) [noise=true , address=45.91.116.9:4271 , piece_id=VZJVLEOMS452TXPJOMIVF4K36YRPLQW3EGZNMEB4BZ7HK7PTDQXQ, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.050521198 (624h48m14.921604233s) [noise=true , address=145.79.227.24:4614 , piece_id=6BJGK6OQWTHZCY3GQSQV7LVJWJOAJSFQSSVPDJ3RGX4EUFIISFRA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.050715528 (624h48m14.921798573s) [noise=true , address=145.79.227.23:4575 , piece_id=NXYLJMIMKJYMJ2UF5BYXYODDUOCROHYNDEHXCO2G5IFVVZQFLXYQ, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.050816169 (624h48m14.921899204s) [noise=true , address=145.79.227.38:4004 , piece_id=OMFDOFGQ3EBWZPON3I3DP6LWBP3BWWEEFHU34EXZBF2BAJLP3TNQ, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.050998136 (624h48m14.922081171s) [noise=true , address=145.79.227.17:4985 , piece_id=Q37B2D24JXNSSJIEIBRSYEE6IEOQV7IG4SEIGLFWLW2PBTWZU5TA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.051085192 (624h48m14.922168227s) [noise=true , address=145.79.227.44:4219 , piece_id=TQUYQDIUJWUX43HZCUOHECBX77YNCL2SZ6BMGX2VQ4T2XUPWNKJA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.051462040 (624h48m14.922545075s) [noise=true , address=77.90.31.17:4788 , piece_id=4G5Z3ECGM5HBH7PELBGGDKXGDPHTNRHKEBXFEKVEP5ZDFXVIGWUA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.051871809 (624h48m14.922954844s) [noise=true , address=77.90.31.16:4760 , piece_id=P56LD3DX5YLE2MPV3RORGLX25G5YDT2TNOVCA5CLGGBYZN6LUUTQ, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.052116004 (624h48m14.923199039s) [noise=true , address=145.79.227.33:4894 , piece_id=MYHRMEIWJDASEUHHTGRKAN5IRNE7QBTO4VNX62LHJVGBIJS5UVLA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:15.052414814 (624h48m14.923497859s) [noise=true , address=145.79.227.19:4087 , piece_id=7EXAHR36NSDTHNUWQQZTIS46XWFQUWTKAEIPTMSQHZHDJHCKI6PA, seg_pos={0 0}, upload=547498]: Uploading piece...
Dec 31 16:17:16.513009118 (624h48m16.384092153s) [upload=547498]: Next returned nil segment
Dec 31 16:17:16.513023495 (624h48m16.38410654s) [upload=547498]: Waiting for error group managing segments...
Dec 31 16:17:16.908124481 (624h48m16.779207526s) [noise=true , address=145.79.227.19:4087 , piece_id=7EXAHR36NSDTHNUWQQZTIS46XWFQUWTKAEIPTMSQHZHDJHCKI6PA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:16.918737279 (624h48m16.789820325s) [noise=true , address=145.79.227.6:4278 , piece_id=UPW54EA7CL2QDHHSBOCOUHOJ4PO664J7HPFRMKDZVRJWLQLZQE3Q, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:16.964651124 (624h48m16.835734159s) [noise=true , address=145.79.227.17:4985 , piece_id=Q37B2D24JXNSSJIEIBRSYEE6IEOQV7IG4SEIGLFWLW2PBTWZU5TA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:16.994270222 (624h48m16.865353258s) [noise=true , address=145.79.227.38:4004 , piece_id=OMFDOFGQ3EBWZPON3I3DP6LWBP3BWWEEFHU34EXZBF2BAJLP3TNQ, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:16.997634448 (624h48m16.868717483s) [noise=true , address=145.79.227.23:4575 , piece_id=NXYLJMIMKJYMJ2UF5BYXYODDUOCROHYNDEHXCO2G5IFVVZQFLXYQ, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.022216365 (624h48m16.8932994s) [noise=true , address=145.79.227.24:4614 , piece_id=6BJGK6OQWTHZCY3GQSQV7LVJWJOAJSFQSSVPDJ3RGX4EUFIISFRA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.036007444 (624h48m16.907090479s) [noise=true , address=5.231.73.3:4030 , piece_id=II77J2BA2EEOJXS4KPGI3GTC4AYQ5Z6NL4ABJCGP2XZEFBP5ABOA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.142349752 (624h48m17.013432787s) [noise=true , address=77.90.31.13:4657 , piece_id=5ITVL3WNXVLJLNGPJS5YSLTOSLULDQGK6F64TJQ6K52MZG2UBIMQ, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.318258862 (624h48m17.189341897s) [noise=true , address=77.90.31.16:4760 , piece_id=P56LD3DX5YLE2MPV3RORGLX25G5YDT2TNOVCA5CLGGBYZN6LUUTQ, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.326042824 (624h48m17.197125859s) [noise=true , address=45.91.116.9:4271 , piece_id=VZJVLEOMS452TXPJOMIVF4K36YRPLQW3EGZNMEB4BZ7HK7PTDQXQ, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.704912834 (624h48m17.575995869s) [noise=true , address=45.91.116.8:4066 , piece_id=DJA5A7MOGJUHX6QKJ32WG3S6HUXE5PXD4DOOMCHZSVZ3FQCFUNRQ, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.808151081 (624h48m17.679234116s) [noise=true , address=145.79.227.35:4492 , piece_id=5KJRZLQ3H3S6SFZWLTKRQU3JIN7RY4E5FJFDDMPHCY6LCYXTVOPA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:17.941220429 (624h48m17.812303464s) [noise=true , address=145.79.227.44:4219 , piece_id=TQUYQDIUJWUX43HZCUOHECBX77YNCL2SZ6BMGX2VQ4T2XUPWNKJA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:18.110531386 (624h48m17.981614421s) [noise=true , address=145.79.227.33:4894 , piece_id=MYHRMEIWJDASEUHHTGRKAN5IRNE7QBTO4VNX62LHJVGBIJS5UVLA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: <nil>
Dec 31 16:17:18.110557937 (624h48m17.981640982s) [seg_pos={0 0}, upload=547498]: Segment reached optimal threshold of 14 pieces.
Dec 31 16:17:18.110785891 (624h48m17.981868956s) [noise=true , address=45.91.116.10:4093 , piece_id=ILTD2BF7GDMGAP7YLYZPEIFHDNX5BJZ5E4KFOUTKEXWF7NNFI64A, seg_pos={0 0}, upload=547498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ): protocol: expected piece hash; context canceled; EOF
Dec 31 16:17:18.110795980 (624h48m17.981879035s) [noise=true , address=45.91.116.10:4093 , piece_id=ILTD2BF7GDMGAP7YLYZPEIFHDNX5BJZ5E4KFOUTKEXWF7NNFI64A, seg_pos={0 0}, upload=547498]: Long tail cancellation due to optimal threshold
Dec 31 16:17:18.110806109 (624h48m17.981889144s) [noise=true , address=77.90.31.12:4606 , piece_id=CMC465DGAG3VRLPTRWL2TNC6MD3HFPIABJJQUTHVSL32LFWPQRBA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pLKmTja7ttT6vqjq8vHMYR9RLfnpqxY9xm9GDBG8ez9aHujs): protocol: write tcp 188.166.225.170:48024->77.90.31.12:4606: use of closed network connection; write tcp 188.166.225.170:48024->77.90.31.12:4606: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48024->77.90.31.12:4606: use of closed network connection
Dec 31 16:17:18.110813563 (624h48m17.981896608s) [noise=true , address=77.90.31.12:4606 , piece_id=CMC465DGAG3VRLPTRWL2TNC6MD3HFPIABJJQUTHVSL32LFWPQRBA, seg_pos={0 0}, upload=547498]: Long tail cancellation due to optimal threshold
Dec 31 16:17:18.111107804 (624h48m17.982190879s) [noise=true , address=45.91.116.7:4027 , piece_id=3AESXZLKXV7X2HH3OJHSDPMDY2OHI2GAFIAAFYJMOVFAJDU2FAUA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF
Dec 31 16:17:18.111117783 (624h48m17.982200828s) [noise=true , address=45.91.116.7:4027 , piece_id=3AESXZLKXV7X2HH3OJHSDPMDY2OHI2GAFIAAFYJMOVFAJDU2FAUA, seg_pos={0 0}, upload=547498]: Long tail cancellation due to optimal threshold
Dec 31 16:17:18.111224586 (624h48m17.982307671s) [noise=true , address=77.90.31.17:4788 , piece_id=4G5Z3ECGM5HBH7PELBGGDKXGDPHTNRHKEBXFEKVEP5ZDFXVIGWUA, seg_pos={0 0}, upload=547498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: read tcp 188.166.225.170:50208->77.90.31.17:4788: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 16:17:18.111233905 (624h48m17.98231694s) [noise=true , address=77.90.31.17:4788 , piece_id=4G5Z3ECGM5HBH7PELBGGDKXGDPHTNRHKEBXFEKVEP5ZDFXVIGWUA, seg_pos={0 0}, upload=547498]: Long tail cancellation due to optimal threshold
Dec 31 16:17:18.111271807 (624h48m17.982354892s) [seg_pos={0 0}, upload=547498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 16:17:18.111364793 (624h48m17.982447838s) [upload=547498]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 16:17:18.111372739 (624h48m17.982455784s) [upload=547498]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 16:17:18.397617841 (624h48m18.268700876s) [upload=547498]: Done uploading segments...
Dec 31 16:17:18.397637810 (624h48m18.268720845s) [upload=547498]: Upload finished. err: <nil>
Duration: 45.189433039s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
6.383343512s trace:
https://jaeger.dev.storj.io/trace/5582e72abe7a8819
FAIL (2025-12-31 16:49:07 UTC)
orbiter warning: context deadline exceeded
Duration: 6.38335866s
edge
gateway
us-select-1
|