Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....X................................X......................] duration:
14.634248806s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:52:30 UTC)
orbiter warning: context deadline exceeded
Duration: 14.634283322s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [....X......XX.X..XXX.X.XXX.XX.XX...X.X........X.X.X.X....XXX] duration:
30.002075668s trace:
https://jaeger.dev.storj.io/trace/6550d4954a6238c8
FAIL (2025-12-28 15:52:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: RJ4HXGLN4FHCKRWIYGVTXXO5ATZBKS3LZ57LBVQOXQYD522EKGEQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: S2X3ZEPKRDG55LZZ7X6YMSZVBJNE4V5I6MA5NUZLAL4W7CS5RIOA) context deadline exceeded
======LOGS======
Dec 28 15:51:55.589680011 (552h22m55.460763056s) [upload=484302]: Starting upload
Dec 28 15:51:55.589738914 (552h22m55.460821949s) [upload=484302]: Done starting upload
Dec 28 15:51:55.589975974 (552h22m55.461059019s) [upload=484302]: Uploading segments...
Dec 28 15:51:55.590273732 (552h22m55.461356767s) [upload=484302]: Got next segment. Inline: false
Dec 28 15:51:55.590313868 (552h22m55.461396903s) [upload=484302]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:51:55.590324748 (552h22m55.461407783s) [upload=484302]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:51:55.836362265 (552h22m55.70744531s) [seg_pos={0 0}, upload=484302]: Begin upload segment...
Dec 28 15:51:55.836421388 (552h22m55.707504423s) [seg_pos={0 0}, upload=484302]: Done begin upload segment.
Dec 28 15:51:55.836455413 (552h22m55.707538458s) [noise=true , address=145.79.227.21:4473 , piece_id=WFY76I7ZIQWBNGM6AMREHJJ7HSVQ45WPBA3FX2DSHWE2MDZBNCPQ, seg_pos={0 0}, upload=484302]: Uploading piece...
Dec 28 15:51:55.836547287 (552h22m55.707630332s) [noise=true , address=145.79.227.51:4410 , piece_id=RHPCDRDCHCGXH255VE6QYWACNKQY5YYBDH2JJWFRPGA7UQKVRWOA, seg_pos={0 0}, upload=484302]: Uploading piece...
Dec 28 15:51:55.837290002 (552h22m55.708373047s) [noise=true , address=145.79.227.36:4517 , piece_id=VGSFWMHFSJVG4LEJXJCWMTOZDGQP5T62TADZ6YXLSU6YPRKTENOQ, seg_pos={0 0}, upload=484302]: Uploading piece...
Dec 28 15:51:55.837430580 (552h22m55.708513625s) [noise=true , address=145.79.227.32:4836 , piece_id=PGP3UFKSKI5TOYIB7NUR2BDOTXZERY3EDTV2YKYROVEQEQ7L64RA, seg_pos={0 0}, upload=484302]: Uploading piece...
Dec 28 15:51:55.837740199 (552h22m55.708823234s) [noise=true , address=145.79.227.2:4114 , piece_id=3YAW7XYHNF6IBHMDOCVBLMTWJ7QO3JTXVBL6PVFZJT5SJHMWYXUA, seg_pos={0 0}, upload=484302]: Uploading piece...
Dec 28 15:51:55.838301548 (552h22m55.709384593s) [noise=true , address=77.90.31.15:4700 , piece_id=C5XNR2OZ3PPDJ3KFXVK532RONNHOSBHOZOFR5LPJIZAA6MBH6N3Q, seg_pos={0 0}, upload=484302]: Uploading piece...
Dec 28 15:51:55.838320344 (552h22m55.709403389s) [noise=true , address=45.91.116.11:4120 , piece_id=S2X3ZEPKRDG55LZZ7X6YMSZVBJNE4V5I6MA5NUZLAL4W7CS5RIOA, seg_pos={0 0}, upload=484302]: Uploading piece...
Dec 28 15:51:55.838874308 (552h22m55.709957354s) [noise=true , address=145.79.227.39:4013 , piece_id=4HRCIQYFXZ5DEYVAIFLRHSFDIUXOXBGUIX5UXP2UBAHS2STGDPGA, seg_pos={0 0}, upload=484302]: Uploading piece...
Dec 28 15:51:55.839055764 (552h22m55.710138809s) [noise=true , address=145.79.227.14:4839 , piece_id=FP2JS7VBPZJHHYXI52JHH6FX3IINV6CJ5BNR56K5ETSM6U3QXTNA, seg_pos={0 0}, upload=484302]: Uploading piece...
Dec 28 15:51:55.839428223 (552h22m55.710511268s) [noise=true , address=45.91.116.7:4011 , piece_id=RJ4HXGLN4FHCKRWIYGVTXXO5ATZBKS3LZ57LBVQOXQYD522EKGEQ, seg_pos={0 0}, upload=484302]: Uploading piece...
Dec 28 15:51:55.839719728 (552h22m55.710802763s) [noise=true , address=145.79.227.44:4195 , piece_id=6ZPLIFTV7CAWQQIFJMS7Q7T52XVLI3OZQIZNWNRXXZCALQU53MEA, seg_pos={0 0}, upload=484302]: Uploading piece...
Dec 28 15:51:55.839876417 (552h22m55.710959462s) [noise=true , address=45.91.116.4:4939 , piece_id=FYKVAJHH42UUQL7MSWFG73XJ3JOGD4PGK3DNTQMBCXFXIWFZIBEQ, seg_pos={0 0}, upload=484302]: Uploading piece...
Dec 28 15:51:55.840327927 (552h22m55.711410972s) [noise=true , address=145.79.227.15:4876 , piece_id=QPUBRDQYTBTLY2QS4G2OSYAMPFNLAK3GNBUZPMOPGZCSESVEXQNQ, seg_pos={0 0}, upload=484302]: Uploading piece...
Dec 28 15:51:55.840346151 (552h22m55.711429196s) [noise=true , address=145.79.227.50:4375 , piece_id=ND5NMHAZPKDLHYUFZJOPFVLAEEEE7MA4OFCRQ27JPDTCJZTOAEEQ, seg_pos={0 0}, upload=484302]: Uploading piece...
Dec 28 15:51:55.840886861 (552h22m55.711969906s) [noise=true , address=145.79.227.52:4446 , piece_id=525EDWO4JWNGIS7D4OU4GIYDFQTOLW4SIDL7Y3BVWT4FF7CJFYGA, seg_pos={0 0}, upload=484302]: Uploading piece...
Dec 28 15:51:55.841968220 (552h22m55.713051265s) [noise=true , address=145.79.227.19:4078 , piece_id=4E4CXFYKLTURU35L5NQ5ZIBAWOD6LJJQ3SQD54IBV3B6EXHKDMAA, seg_pos={0 0}, upload=484302]: Uploading piece...
Dec 28 15:51:57.536601676 (552h22m57.407684711s) [upload=484302]: Next returned nil segment
Dec 28 15:51:57.536615864 (552h22m57.407698909s) [upload=484302]: Waiting for error group managing segments...
Dec 28 15:51:57.848168492 (552h22m57.719251527s) [noise=true , address=145.79.227.21:4473 , piece_id=WFY76I7ZIQWBNGM6AMREHJJ7HSVQ45WPBA3FX2DSHWE2MDZBNCPQ, seg_pos={0 0}, upload=484302]: Done uploading piece. err: <nil>
Dec 28 15:51:57.852408275 (552h22m57.723491321s) [noise=true , address=145.79.227.32:4836 , piece_id=PGP3UFKSKI5TOYIB7NUR2BDOTXZERY3EDTV2YKYROVEQEQ7L64RA, seg_pos={0 0}, upload=484302]: Done uploading piece. err: <nil>
Dec 28 15:51:57.855930101 (552h22m57.727013137s) [noise=true , address=145.79.227.14:4839 , piece_id=FP2JS7VBPZJHHYXI52JHH6FX3IINV6CJ5BNR56K5ETSM6U3QXTNA, seg_pos={0 0}, upload=484302]: Done uploading piece. err: <nil>
Dec 28 15:51:57.859144342 (552h22m57.730227387s) [noise=true , address=145.79.227.52:4446 , piece_id=525EDWO4JWNGIS7D4OU4GIYDFQTOLW4SIDL7Y3BVWT4FF7CJFYGA, seg_pos={0 0}, upload=484302]: Done uploading piece. err: <nil>
Dec 28 15:51:57.859313524 (552h22m57.730396559s) [noise=true , address=145.79.227.50:4375 , piece_id=ND5NMHAZPKDLHYUFZJOPFVLAEEEE7MA4OFCRQ27JPDTCJZTOAEEQ, seg_pos={0 0}, upload=484302]: Done uploading piece. err: <nil>
Dec 28 15:51:57.859954374 (552h22m57.731037409s) [noise=true , address=145.79.227.2:4114 , piece_id=3YAW7XYHNF6IBHMDOCVBLMTWJ7QO3JTXVBL6PVFZJT5SJHMWYXUA, seg_pos={0 0}, upload=484302]: Done uploading piece. err: <nil>
Dec 28 15:51:57.862825672 (552h22m57.733908717s) [noise=true , address=145.79.227.15:4876 , piece_id=QPUBRDQYTBTLY2QS4G2OSYAMPFNLAK3GNBUZPMOPGZCSESVEXQNQ, seg_pos={0 0}, upload=484302]: Done uploading piece. err: <nil>
Dec 28 15:51:57.867382349 (552h22m57.738465384s) [noise=true , address=145.79.227.19:4078 , piece_id=4E4CXFYKLTURU35L5NQ5ZIBAWOD6LJJQ3SQD54IBV3B6EXHKDMAA, seg_pos={0 0}, upload=484302]: Done uploading piece. err: <nil>
Dec 28 15:51:57.867804482 (552h22m57.738887527s) [noise=true , address=145.79.227.51:4410 , piece_id=RHPCDRDCHCGXH255VE6QYWACNKQY5YYBDH2JJWFRPGA7UQKVRWOA, seg_pos={0 0}, upload=484302]: Done uploading piece. err: <nil>
Dec 28 15:51:57.870178813 (552h22m57.741261848s) [noise=true , address=145.79.227.36:4517 , piece_id=VGSFWMHFSJVG4LEJXJCWMTOZDGQP5T62TADZ6YXLSU6YPRKTENOQ, seg_pos={0 0}, upload=484302]: Done uploading piece. err: <nil>
Dec 28 15:51:57.870746133 (552h22m57.741829178s) [noise=true , address=145.79.227.39:4013 , piece_id=4HRCIQYFXZ5DEYVAIFLRHSFDIUXOXBGUIX5UXP2UBAHS2STGDPGA, seg_pos={0 0}, upload=484302]: Done uploading piece. err: <nil>
Dec 28 15:51:58.059438966 (552h22m57.930522001s) [noise=true , address=45.91.116.11:4120 , piece_id=S2X3ZEPKRDG55LZZ7X6YMSZVBJNE4V5I6MA5NUZLAL4W7CS5RIOA, seg_pos={0 0}, upload=484302]: Done uploading piece. err: <nil>
Dec 28 15:51:58.182756614 (552h22m58.053839649s) [noise=true , address=45.91.116.7:4011 , piece_id=RJ4HXGLN4FHCKRWIYGVTXXO5ATZBKS3LZ57LBVQOXQYD522EKGEQ, seg_pos={0 0}, upload=484302]: Done uploading piece. err: <nil>
Dec 28 15:51:58.182777654 (552h22m58.053860699s) [seg_pos={0 0}, upload=484302]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:51:58.182978726 (552h22m58.054061762s) [noise=true , address=45.91.116.4:4939 , piece_id=FYKVAJHH42UUQL7MSWFG73XJ3JOGD4PGK3DNTQMBCXFXIWFZIBEQ, seg_pos={0 0}, upload=484302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 28 15:51:58.182985941 (552h22m58.054068976s) [noise=true , address=45.91.116.4:4939 , piece_id=FYKVAJHH42UUQL7MSWFG73XJ3JOGD4PGK3DNTQMBCXFXIWFZIBEQ, seg_pos={0 0}, upload=484302]: Long tail cancellation due to optimal threshold
Dec 28 15:51:58.183217060 (552h22m58.054300106s) [noise=true , address=145.79.227.44:4195 , piece_id=6ZPLIFTV7CAWQQIFJMS7Q7T52XVLI3OZQIZNWNRXXZCALQU53MEA, seg_pos={0 0}, upload=484302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 28 15:51:58.183234083 (552h22m58.054317118s) [noise=true , address=145.79.227.44:4195 , piece_id=6ZPLIFTV7CAWQQIFJMS7Q7T52XVLI3OZQIZNWNRXXZCALQU53MEA, seg_pos={0 0}, upload=484302]: Long tail cancellation due to optimal threshold
Dec 28 15:51:58.183251276 (552h22m58.054334311s) [noise=true , address=77.90.31.15:4700 , piece_id=C5XNR2OZ3PPDJ3KFXVK532RONNHOSBHOZOFR5LPJIZAA6MBH6N3Q, seg_pos={0 0}, upload=484302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: read tcp 188.166.225.170:34422->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 15:51:58.183258329 (552h22m58.054341374s) [noise=true , address=77.90.31.15:4700 , piece_id=C5XNR2OZ3PPDJ3KFXVK532RONNHOSBHOZOFR5LPJIZAA6MBH6N3Q, seg_pos={0 0}, upload=484302]: Long tail cancellation due to optimal threshold
Dec 28 15:51:58.183276494 (552h22m58.054359529s) [seg_pos={0 0}, upload=484302]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:51:58.183350324 (552h22m58.054433359s) [upload=484302]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:51:58.183354923 (552h22m58.054437958s) [upload=484302]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:51:58.502665995 (552h22m58.37374903s) [upload=484302]: Done uploading segments...
Dec 28 15:51:58.502684941 (552h22m58.373767976s) [upload=484302]: Upload finished. err: <nil>
Duration: 30.002195257s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
5.913998327s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 15:52:22 UTC)
orbiter warning: context deadline exceeded
Duration: 5.914026921s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....X................................X.....................] duration:
13.430279545s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:52:07 UTC)
orbiter warning: context deadline exceeded
Duration: 13.430294974s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [............X....X...X........................X.............] duration:
37.249039546s trace:
https://jaeger.dev.storj.io/trace/3cc795a000779b86
FAIL (2025-12-28 15:51:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ) context deadline exceeded; piecestore: (Node ID: 128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, Piece ID: GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q) context deadline exceeded; piecestore: (Node ID: 12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, Piece ID: TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A) context deadline exceeded
======LOGS======
48.156469147 (552h11m48.027552193s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157156416 (552h11m48.028239461s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157556748 (552h11m48.028639793s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157945468 (552h11m48.029028503s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158645151 (552h11m48.029728196s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158842136 (552h11m48.029925171s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158983275 (552h11m48.03006631s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159344152 (552h11m48.030427187s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159350725 (552h11m48.03043376s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159710760 (552h11m48.030793795s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159732391 (552h11m48.030815437s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160068310 (552h11m48.031151346s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160439438 (552h11m48.031522473s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160813179 (552h11m48.031896225s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160839249 (552h11m48.031922284s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161271813 (552h11m48.032354848s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161765813 (552h11m48.032848848s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:49.456221594 (552h11m49.327304629s) [upload=484135]: Next returned nil segment
Dec 28 15:40:49.456235460 (552h11m49.327318495s) [upload=484135]: Waiting for error group managing segments...
Dec 28 15:40:49.789381987 (552h11m49.660465032s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.818159755 (552h11m49.6892428s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.901730607 (552h11m49.772813642s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.297735267 (552h11m50.168818312s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.866598958 (552h11m50.737682003s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.955467169 (552h11m50.826550214s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.029562809 (552h11m50.900645854s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.033202229 (552h11m50.904285264s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.119305864 (552h11m50.990388909s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.271054029 (552h11m51.142137074s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.374827580 (552h11m51.245910626s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.380607978 (552h11m51.251691013s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.500440018 (552h11m51.371523063s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688250240 (552h11m51.559333285s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688275438 (552h11m51.559358473s) [seg_pos={0 0}, upload=484135]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:40:51.688501768 (552h11m51.559584804s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688512109 (552h11m51.559595144s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688551694 (552h11m51.559634729s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688560741 (552h11m51.559643786s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688768767 (552h11m51.559851802s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWqAGD7Vrj2yzEVyNbfPp278Gga42xPLyAcdzGvUSuz3EgJYh): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688776031 (552h11m51.559859066s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688811729 (552h11m51.559894784s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688820707 (552h11m51.559903752s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688840654 (552h11m51.559923689s) [seg_pos={0 0}, upload=484135]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:40:51.688958800 (552h11m51.560041845s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:40:51.688965582 (552h11m51.560048617s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:40:51.990040599 (552h11m51.861123664s) [upload=484135]: Done uploading segments...
Dec 28 15:40:51.990091757 (552h11m51.861174802s) [upload=484135]: Upload finished. err: <nil>
Duration: 37.249055778s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
38 out of 60 is failed (threshold=6) [XXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX..XX...XXX.....XX..XX] duration:
5.001411445s trace:
https://jaeger.dev.storj.io/trace/60cb23b4b6763c34
FAIL (2025-12-28 15:51:48 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001435632s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
5.899250511s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 15:51:38 UTC)
orbiter warning: context deadline exceeded
Duration: 5.899279677s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
38 out of 60 is failed (threshold=6) [XXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX..XX...XXX.....XX..X] duration:
5.00170203s trace:
https://jaeger.dev.storj.io/trace/3ed61f91e63e3e17
FAIL (2025-12-28 15:51:29 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001739562s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X................................X....................] duration:
12.749712984s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:51:27 UTC)
orbiter warning: context deadline exceeded
Duration: 12.749737621s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [.....X......XX.X..XXX.X.XXX.XX.XX...X.X........X.X.X.X....XX] duration:
30.002869083s trace:
https://jaeger.dev.storj.io/trace/65cf1b499f84820
FAIL (2025-12-28 15:51:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: ZBEAC33YLBLBKN5PDIJOD665KZPEVIBNJ33M2DOECYTSJFP3ONLA) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: UT4OEYZPRKRZD4Z4OIPT3JKHCS6AKRHOQ3VI3KZWITLG4PGXDF7Q) context deadline exceeded
======LOGS======
15:50:57.098264864 (552h21m56.969347909s) [upload=484286]: Starting upload
Dec 28 15:50:57.098447401 (552h21m56.969530456s) [upload=484286]: Done starting upload
Dec 28 15:50:57.098534377 (552h21m56.969617412s) [upload=484286]: Uploading segments...
Dec 28 15:50:57.098693730 (552h21m56.969776775s) [upload=484286]: Got next segment. Inline: false
Dec 28 15:50:57.098749647 (552h21m56.969832702s) [upload=484286]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:50:57.098756600 (552h21m56.969839645s) [upload=484286]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:50:57.346413691 (552h21m57.217496756s) [seg_pos={0 0}, upload=484286]: Begin upload segment...
Dec 28 15:50:57.346483003 (552h21m57.217566038s) [seg_pos={0 0}, upload=484286]: Done begin upload segment.
Dec 28 15:50:57.346507460 (552h21m57.217590505s) [noise=true , address=145.79.227.48:4302 , piece_id=ETEKXKK4CYJQWT5GPKGO7CA4466LG77Q6EL7HTEH7XBOCHEE7BOA, seg_pos={0 0}, upload=484286]: Uploading piece...
Dec 28 15:50:57.346526937 (552h21m57.217609972s) [noise=true , address=145.79.227.5:4221 , piece_id=VXHMMNCZ2LVFUPF5KGBFKUQ7OZSIAOORMT2WDW7GKPXOBAKQL5EA, seg_pos={0 0}, upload=484286]: Uploading piece...
Dec 28 15:50:57.346581631 (552h21m57.217664666s) [noise=true , address=77.90.31.15:4697 , piece_id=EYNPYCGWT52NZ44ULQCTD5DIVJ4JQ25ZUOAICYVZ3E4EHNQH7YGQ, seg_pos={0 0}, upload=484286]: Uploading piece...
Dec 28 15:50:57.346903805 (552h21m57.21798685s) [noise=true , address=145.79.227.46:4603 , piece_id=QA2BQ7WPZFZNGDQHZLTKGSBVL2OF6WCTFXWLEMH3HM6WDOHGDA6A, seg_pos={0 0}, upload=484286]: Uploading piece...
Dec 28 15:50:57.347204688 (552h21m57.218287723s) [noise=true , address=45.91.116.5:4300 , piece_id=ZBEAC33YLBLBKN5PDIJOD665KZPEVIBNJ33M2DOECYTSJFP3ONLA, seg_pos={0 0}, upload=484286]: Uploading piece...
Dec 28 15:50:57.347484521 (552h21m57.218567566s) [noise=true , address=145.79.227.39:4015 , piece_id=7XP23UL7VTMZFAZ33R677QAAASHC5JFE2CNDHOLTFV7DQNM2D5BA, seg_pos={0 0}, upload=484286]: Uploading piece...
Dec 28 15:50:57.347836150 (552h21m57.218919185s) [noise=true , address=45.91.116.10:4082 , piece_id=DUUVUCO3IMFBWUUPNTNTQOQOB6ZYHVTW67DI2AYOCCQK4JX6VMTA, seg_pos={0 0}, upload=484286]: Uploading piece...
Dec 28 15:50:57.348230561 (552h21m57.219313606s) [noise=true , address=145.79.227.13:4803 , piece_id=EEZ3QAFBW3MAVT2BBXYKSECFH3UYXM6TK3ZKPUOSN7BJAZYOZYFQ, seg_pos={0 0}, upload=484286]: Uploading piece...
Dec 28 15:50:57.348589524 (552h21m57.219672559s) [noise=true , address=145.79.227.37:4567 , piece_id=TKWJK7IDRYWRYFTN7ISG3FVA7RHQV4347OD5UINYXYI433WLKG4A, seg_pos={0 0}, upload=484286]: Uploading piece...
Dec 28 15:50:57.348967153 (552h21m57.220050188s) [noise=true , address=45.91.116.14:4190 , piece_id=VSUEZ2IRXLZFUFXMU6YM2NMWHAPKZ36P2DNZWKNIGQRRHJBUQWXA, seg_pos={0 0}, upload=484286]: Uploading piece...
Dec 28 15:50:57.349313182 (552h21m57.220396227s) [noise=true , address=77.90.31.12:4590 , piece_id=XD4INMJTFVRG3DRH2VLQVRCPP7HIW6NEHG6UBBGB732NB2C7CHCQ, seg_pos={0 0}, upload=484286]: Uploading piece...
Dec 28 15:50:57.350532143 (552h21m57.221615178s) [noise=true , address=145.79.227.7:4295 , piece_id=4M5BAQD3V7PBBFLINN5XHATTU2XJG3CSX56SOGBVOJH5KHP735EQ, seg_pos={0 0}, upload=484286]: Uploading piece...
Dec 28 15:50:57.350554426 (552h21m57.221637471s) [noise=true , address=145.79.227.32:4834 , piece_id=5Y4N7K5VMHN2JYRJP25XJNQIK63XTTQQQU4V6EUV3AW233ARFB7Q, seg_pos={0 0}, upload=484286]: Uploading piece...
Dec 28 15:50:57.351901892 (552h21m57.222984927s) [noise=true , address=145.79.227.51:4409 , piece_id=B2XDBNDN5BRGKXTV2JHSSZAUA5QTAARD2NPHI7REUP6VIHETDLDA, seg_pos={0 0}, upload=484286]: Uploading piece...
Dec 28 15:50:57.353170527 (552h21m57.224253563s) [noise=true , address=45.91.116.4:4937 , piece_id=PMDNYVYIF57EF5L2XJYYBM42C52I2VN4U6JFRG6UR3ZPHVKFGHUQ, seg_pos={0 0}, upload=484286]: Uploading piece...
Dec 28 15:50:57.356429894 (552h21m57.227512929s) [noise=true , address=77.90.31.10:4518 , piece_id=UT4OEYZPRKRZD4Z4OIPT3JKHCS6AKRHOQ3VI3KZWITLG4PGXDF7Q, seg_pos={0 0}, upload=484286]: Uploading piece...
Dec 28 15:50:59.070622545 (552h21m58.94170559s) [upload=484286]: Next returned nil segment
Dec 28 15:50:59.070639897 (552h21m58.941722932s) [upload=484286]: Waiting for error group managing segments...
Dec 28 15:50:59.352190297 (552h21m59.223273342s) [noise=true , address=145.79.227.48:4302 , piece_id=ETEKXKK4CYJQWT5GPKGO7CA4466LG77Q6EL7HTEH7XBOCHEE7BOA, seg_pos={0 0}, upload=484286]: Done uploading piece. err: <nil>
Dec 28 15:50:59.352240883 (552h21m59.223323918s) [noise=true , address=145.79.227.13:4803 , piece_id=EEZ3QAFBW3MAVT2BBXYKSECFH3UYXM6TK3ZKPUOSN7BJAZYOZYFQ, seg_pos={0 0}, upload=484286]: Done uploading piece. err: <nil>
Dec 28 15:50:59.359368555 (552h21m59.23045161s) [noise=true , address=145.79.227.46:4603 , piece_id=QA2BQ7WPZFZNGDQHZLTKGSBVL2OF6WCTFXWLEMH3HM6WDOHGDA6A, seg_pos={0 0}, upload=484286]: Done uploading piece. err: <nil>
Dec 28 15:50:59.359727139 (552h21m59.230810174s) [noise=true , address=145.79.227.39:4015 , piece_id=7XP23UL7VTMZFAZ33R677QAAASHC5JFE2CNDHOLTFV7DQNM2D5BA, seg_pos={0 0}, upload=484286]: Done uploading piece. err: <nil>
Dec 28 15:50:59.364680370 (552h21m59.235763405s) [noise=true , address=145.79.227.37:4567 , piece_id=TKWJK7IDRYWRYFTN7ISG3FVA7RHQV4347OD5UINYXYI433WLKG4A, seg_pos={0 0}, upload=484286]: Done uploading piece. err: <nil>
Dec 28 15:50:59.374505669 (552h21m59.245588714s) [noise=true , address=145.79.227.5:4221 , piece_id=VXHMMNCZ2LVFUPF5KGBFKUQ7OZSIAOORMT2WDW7GKPXOBAKQL5EA, seg_pos={0 0}, upload=484286]: Done uploading piece. err: <nil>
Dec 28 15:50:59.381774429 (552h21m59.252857474s) [noise=true , address=145.79.227.7:4295 , piece_id=4M5BAQD3V7PBBFLINN5XHATTU2XJG3CSX56SOGBVOJH5KHP735EQ, seg_pos={0 0}, upload=484286]: Done uploading piece. err: <nil>
Dec 28 15:50:59.385531554 (552h21m59.256614599s) [noise=true , address=145.79.227.51:4409 , piece_id=B2XDBNDN5BRGKXTV2JHSSZAUA5QTAARD2NPHI7REUP6VIHETDLDA, seg_pos={0 0}, upload=484286]: Done uploading piece. err: <nil>
Dec 28 15:50:59.395684176 (552h21m59.266767211s) [noise=true , address=145.79.227.32:4834 , piece_id=5Y4N7K5VMHN2JYRJP25XJNQIK63XTTQQQU4V6EUV3AW233ARFB7Q, seg_pos={0 0}, upload=484286]: Done uploading piece. err: <nil>
Dec 28 15:50:59.556628031 (552h21m59.427711066s) [noise=true , address=77.90.31.10:4518 , piece_id=UT4OEYZPRKRZD4Z4OIPT3JKHCS6AKRHOQ3VI3KZWITLG4PGXDF7Q, seg_pos={0 0}, upload=484286]: Done uploading piece. err: <nil>
Dec 28 15:50:59.674123291 (552h21m59.545206336s) [noise=true , address=77.90.31.15:4697 , piece_id=EYNPYCGWT52NZ44ULQCTD5DIVJ4JQ25ZUOAICYVZ3E4EHNQH7YGQ, seg_pos={0 0}, upload=484286]: Done uploading piece. err: <nil>
Dec 28 15:50:59.686191721 (552h21m59.557274767s) [noise=true , address=45.91.116.4:4937 , piece_id=PMDNYVYIF57EF5L2XJYYBM42C52I2VN4U6JFRG6UR3ZPHVKFGHUQ, seg_pos={0 0}, upload=484286]: Done uploading piece. err: <nil>
Dec 28 15:50:59.687024437 (552h21m59.558107472s) [noise=true , address=45.91.116.5:4300 , piece_id=ZBEAC33YLBLBKN5PDIJOD665KZPEVIBNJ33M2DOECYTSJFP3ONLA, seg_pos={0 0}, upload=484286]: Done uploading piece. err: <nil>
Dec 28 15:50:59.687034586 (552h21m59.558117621s) [seg_pos={0 0}, upload=484286]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:50:59.687225260 (552h21m59.558308305s) [noise=true , address=77.90.31.12:4590 , piece_id=XD4INMJTFVRG3DRH2VLQVRCPP7HIW6NEHG6UBBGB732NB2C7CHCQ, seg_pos={0 0}, upload=484286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 28 15:50:59.687231381 (552h21m59.558314416s) [noise=true , address=77.90.31.12:4590 , piece_id=XD4INMJTFVRG3DRH2VLQVRCPP7HIW6NEHG6UBBGB732NB2C7CHCQ, seg_pos={0 0}, upload=484286]: Long tail cancellation due to optimal threshold
Dec 28 15:50:59.687458964 (552h21m59.558541999s) [noise=true , address=45.91.116.10:4082 , piece_id=DUUVUCO3IMFBWUUPNTNTQOQOB6ZYHVTW67DI2AYOCCQK4JX6VMTA, seg_pos={0 0}, upload=484286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF
Dec 28 15:50:59.687465367 (552h21m59.558548412s) [noise=true , address=45.91.116.10:4082 , piece_id=DUUVUCO3IMFBWUUPNTNTQOQOB6ZYHVTW67DI2AYOCCQK4JX6VMTA, seg_pos={0 0}, upload=484286]: Long tail cancellation due to optimal threshold
Dec 28 15:50:59.687635411 (552h21m59.558718446s) [noise=true , address=45.91.116.14:4190 , piece_id=VSUEZ2IRXLZFUFXMU6YM2NMWHAPKZ36P2DNZWKNIGQRRHJBUQWXA, seg_pos={0 0}, upload=484286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:47544->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:47544->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47544->45.91.116.14:4190: use of closed network connection
Dec 28 15:50:59.687642073 (552h21m59.558725109s) [noise=true , address=45.91.116.14:4190 , piece_id=VSUEZ2IRXLZFUFXMU6YM2NMWHAPKZ36P2DNZWKNIGQRRHJBUQWXA, seg_pos={0 0}, upload=484286]: Long tail cancellation due to optimal threshold
Dec 28 15:50:59.687657663 (552h21m59.558740698s) [seg_pos={0 0}, upload=484286]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:50:59.687728658 (552h21m59.558811703s) [upload=484286]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:50:59.687734890 (552h21m59.558817935s) [upload=484286]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:50:59.982663613 (552h21m59.853746648s) [upload=484286]: Done uploading segments...
Dec 28 15:50:59.982682790 (552h21m59.853765835s) [upload=484286]: Upload finished. err: <nil>
Duration: 30.002893289s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.............X....X...X........................X............] duration:
40.931856087s trace:
https://jaeger.dev.storj.io/trace/3cc795a000779b86
FAIL (2025-12-28 15:51:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ) context deadline exceeded; piecestore: (Node ID: 128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, Piece ID: GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q) context deadline exceeded; piecestore: (Node ID: 12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, Piece ID: TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A) context deadline exceeded
======LOGS======
48.156469147 (552h11m48.027552193s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157156416 (552h11m48.028239461s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157556748 (552h11m48.028639793s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157945468 (552h11m48.029028503s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158645151 (552h11m48.029728196s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158842136 (552h11m48.029925171s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158983275 (552h11m48.03006631s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159344152 (552h11m48.030427187s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159350725 (552h11m48.03043376s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159710760 (552h11m48.030793795s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159732391 (552h11m48.030815437s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160068310 (552h11m48.031151346s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160439438 (552h11m48.031522473s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160813179 (552h11m48.031896225s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160839249 (552h11m48.031922284s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161271813 (552h11m48.032354848s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161765813 (552h11m48.032848848s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:49.456221594 (552h11m49.327304629s) [upload=484135]: Next returned nil segment
Dec 28 15:40:49.456235460 (552h11m49.327318495s) [upload=484135]: Waiting for error group managing segments...
Dec 28 15:40:49.789381987 (552h11m49.660465032s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.818159755 (552h11m49.6892428s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.901730607 (552h11m49.772813642s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.297735267 (552h11m50.168818312s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.866598958 (552h11m50.737682003s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.955467169 (552h11m50.826550214s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.029562809 (552h11m50.900645854s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.033202229 (552h11m50.904285264s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.119305864 (552h11m50.990388909s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.271054029 (552h11m51.142137074s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.374827580 (552h11m51.245910626s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.380607978 (552h11m51.251691013s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.500440018 (552h11m51.371523063s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688250240 (552h11m51.559333285s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688275438 (552h11m51.559358473s) [seg_pos={0 0}, upload=484135]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:40:51.688501768 (552h11m51.559584804s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688512109 (552h11m51.559595144s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688551694 (552h11m51.559634729s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688560741 (552h11m51.559643786s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688768767 (552h11m51.559851802s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWqAGD7Vrj2yzEVyNbfPp278Gga42xPLyAcdzGvUSuz3EgJYh): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688776031 (552h11m51.559859066s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688811729 (552h11m51.559894784s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688820707 (552h11m51.559903752s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688840654 (552h11m51.559923689s) [seg_pos={0 0}, upload=484135]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:40:51.688958800 (552h11m51.560041845s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:40:51.688965582 (552h11m51.560048617s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:40:51.990040599 (552h11m51.861123664s) [upload=484135]: Done uploading segments...
Dec 28 15:40:51.990091757 (552h11m51.861174802s) [upload=484135]: Upload finished. err: <nil>
Duration: 40.931878489s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
38 out of 60 is failed (threshold=6) [XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX..XX...XXX.....XX..] duration:
3.518118719s trace:
https://jaeger.dev.storj.io/trace/721ddd16157f1400
FAIL (2025-12-28 15:51:09 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.51814607s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [X.....X......XX.X..XXX.X.XXX.XX.XX...X.X........X.X.X.X....X] duration:
30.003040828s trace:
https://jaeger.dev.storj.io/trace/46bd1043865c320
FAIL (2025-12-28 15:50:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 7IPD4PRMZXZ26LHAFV3XHPNMG2EHLZ5DLIOIAXFLE7TKIEP3HLCA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: 5AAADVJS2KFHLTX2U7XWRFG7GZUT6JRC43T6NEE3KCQOSCZHJBYQ) context deadline exceeded
======LOGS======
egments...
Dec 28 15:50:23.360504012 (552h21m23.231587057s) [upload=484278]: Got next segment. Inline: false
Dec 28 15:50:23.360554568 (552h21m23.231637603s) [upload=484278]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:50:23.360559788 (552h21m23.231642823s) [upload=484278]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:50:23.608762147 (552h21m23.479845192s) [seg_pos={0 0}, upload=484278]: Begin upload segment...
Dec 28 15:50:23.608832080 (552h21m23.479915115s) [seg_pos={0 0}, upload=484278]: Done begin upload segment.
Dec 28 15:50:23.608895030 (552h21m23.479978065s) [noise=true , address=145.79.227.36:4519 , piece_id=KB4O5C2EXCMPUHYXO6MR5JYH2SVO4DGMMNXN3IGXE2PIKXIIZJSA, seg_pos={0 0}, upload=484278]: Uploading piece...
Dec 28 15:50:23.610002068 (552h21m23.481085103s) [noise=true , address=145.79.227.16:4910 , piece_id=BIZQPWZZWJCP46AJSLLTE4O3CSZSBYGMFOKCS4NLSBDV2C7OHB3Q, seg_pos={0 0}, upload=484278]: Uploading piece...
Dec 28 15:50:23.610008170 (552h21m23.481091205s) [noise=true , address=145.79.227.3:4149 , piece_id=DY6VYA2LSY74G7KQVGDBYPPWISISJ3775SYWK27CN7VAQKXCXZYA, seg_pos={0 0}, upload=484278]: Uploading piece...
Dec 28 15:50:23.610324232 (552h21m23.481407277s) [noise=true , address=145.79.227.42:4122 , piece_id=FOTNUWNHVC42BFB2GGRXT7MOPUBS7YXXTS2WUJGAXE5P6CE2HFBA, seg_pos={0 0}, upload=484278]: Uploading piece...
Dec 28 15:50:23.610738280 (552h21m23.481821325s) [noise=true , address=77.90.31.10:4518 , piece_id=7IPD4PRMZXZ26LHAFV3XHPNMG2EHLZ5DLIOIAXFLE7TKIEP3HLCA, seg_pos={0 0}, upload=484278]: Uploading piece...
Dec 28 15:50:23.610821379 (552h21m23.481904414s) [noise=true , address=45.91.116.12:4226 , piece_id=VVYSNSK2C25K7WW2274OE5CUWCVD4HCIL6BROEKVVZG2AEC3U7PQ, seg_pos={0 0}, upload=484278]: Uploading piece...
Dec 28 15:50:23.611168950 (552h21m23.482251985s) [noise=true , address=45.91.116.8:4045 , piece_id=MUMTG4I7R3U3OGCQGEO5N2YDSIWDWVR6WDBBHL3ATSSEQOEDMREA, seg_pos={0 0}, upload=484278]: Uploading piece...
Dec 28 15:50:23.611407033 (552h21m23.482490078s) [noise=true , address=45.91.116.11:4120 , piece_id=TQGAJ6MQA4GGNIZ33N56JMR4XN3GNOIZLKPV6XEX4K5EPVXS5D2A, seg_pos={0 0}, upload=484278]: Uploading piece...
Dec 28 15:50:23.611578360 (552h21m23.482661395s) [noise=true , address=145.79.227.17:4972 , piece_id=5R73WGFQJNDCWIW2LV4DASAOMBNHFEFJ5PL45UUC3JRLO4HA4FMA, seg_pos={0 0}, upload=484278]: Uploading piece...
Dec 28 15:50:23.611905804 (552h21m23.482988839s) [noise=true , address=145.79.227.11:4731 , piece_id=F73IPVOFN6TNKT5XZBIOZRXEXCVYTZYIK6RF6DGZAFLNEVNUG6OQ, seg_pos={0 0}, upload=484278]: Uploading piece...
Dec 28 15:50:23.612225813 (552h21m23.483308848s) [noise=true , address=145.79.227.32:4834 , piece_id=RQ6RGHKAI5LKKPXZ6JYHIJH23GWMTWUOOI7WIPHNMESO63GYT6LA, seg_pos={0 0}, upload=484278]: Uploading piece...
Dec 28 15:50:23.612258615 (552h21m23.48334166s) [noise=true , address=45.91.116.4:4940 , piece_id=IDCU4MCBZURJU7GA7N2HYYSCANAA2FHV44RVUX2FSG7EIIXWEF5Q, seg_pos={0 0}, upload=484278]: Uploading piece...
Dec 28 15:50:23.612510154 (552h21m23.483593199s) [noise=true , address=145.79.227.45:4230 , piece_id=S6DWHBSYGJACVDHMXKSDCFEDVKSICXJQQZDUSCRJGCZICVGOY3YA, seg_pos={0 0}, upload=484278]: Uploading piece...
Dec 28 15:50:23.612697991 (552h21m23.483781026s) [noise=true , address=77.90.31.17:4770 , piece_id=5AAADVJS2KFHLTX2U7XWRFG7GZUT6JRC43T6NEE3KCQOSCZHJBYQ, seg_pos={0 0}, upload=484278]: Uploading piece...
Dec 28 15:50:23.613204436 (552h21m23.484287481s) [noise=true , address=145.79.227.19:4077 , piece_id=GRP7AJPLCPA4UTHDTBPIZKAR347257MJF7M47BZX2T22RJZR6ZYQ, seg_pos={0 0}, upload=484278]: Uploading piece...
Dec 28 15:50:23.613721521 (552h21m23.484804566s) [noise=true , address=145.79.227.18:4007 , piece_id=IG2TJ5K7S5NNKKGC3YLYKHS6NTJNVZVFNW72YIGFEXGJJEJ6NAXQ, seg_pos={0 0}, upload=484278]: Uploading piece...
Dec 28 15:50:25.238308557 (552h21m25.109391602s) [upload=484278]: Next returned nil segment
Dec 28 15:50:25.238325800 (552h21m25.109408845s) [upload=484278]: Waiting for error group managing segments...
Dec 28 15:50:25.623270890 (552h21m25.494353935s) [noise=true , address=145.79.227.19:4077 , piece_id=GRP7AJPLCPA4UTHDTBPIZKAR347257MJF7M47BZX2T22RJZR6ZYQ, seg_pos={0 0}, upload=484278]: Done uploading piece. err: <nil>
Dec 28 15:50:25.629772991 (552h21m25.500856026s) [noise=true , address=145.79.227.3:4149 , piece_id=DY6VYA2LSY74G7KQVGDBYPPWISISJ3775SYWK27CN7VAQKXCXZYA, seg_pos={0 0}, upload=484278]: Done uploading piece. err: <nil>
Dec 28 15:50:25.631009876 (552h21m25.502092911s) [noise=true , address=145.79.227.42:4122 , piece_id=FOTNUWNHVC42BFB2GGRXT7MOPUBS7YXXTS2WUJGAXE5P6CE2HFBA, seg_pos={0 0}, upload=484278]: Done uploading piece. err: <nil>
Dec 28 15:50:25.633261153 (552h21m25.504344198s) [noise=true , address=145.79.227.36:4519 , piece_id=KB4O5C2EXCMPUHYXO6MR5JYH2SVO4DGMMNXN3IGXE2PIKXIIZJSA, seg_pos={0 0}, upload=484278]: Done uploading piece. err: <nil>
Dec 28 15:50:25.633904499 (552h21m25.504987534s) [noise=true , address=145.79.227.16:4910 , piece_id=BIZQPWZZWJCP46AJSLLTE4O3CSZSBYGMFOKCS4NLSBDV2C7OHB3Q, seg_pos={0 0}, upload=484278]: Done uploading piece. err: <nil>
Dec 28 15:50:25.635819916 (552h21m25.506902961s) [noise=true , address=145.79.227.45:4230 , piece_id=S6DWHBSYGJACVDHMXKSDCFEDVKSICXJQQZDUSCRJGCZICVGOY3YA, seg_pos={0 0}, upload=484278]: Done uploading piece. err: <nil>
Dec 28 15:50:25.638683939 (552h21m25.509766974s) [noise=true , address=145.79.227.17:4972 , piece_id=5R73WGFQJNDCWIW2LV4DASAOMBNHFEFJ5PL45UUC3JRLO4HA4FMA, seg_pos={0 0}, upload=484278]: Done uploading piece. err: <nil>
Dec 28 15:50:25.639202096 (552h21m25.510285131s) [noise=true , address=145.79.227.11:4731 , piece_id=F73IPVOFN6TNKT5XZBIOZRXEXCVYTZYIK6RF6DGZAFLNEVNUG6OQ, seg_pos={0 0}, upload=484278]: Done uploading piece. err: <nil>
Dec 28 15:50:25.642096598 (552h21m25.513179633s) [noise=true , address=145.79.227.18:4007 , piece_id=IG2TJ5K7S5NNKKGC3YLYKHS6NTJNVZVFNW72YIGFEXGJJEJ6NAXQ, seg_pos={0 0}, upload=484278]: Done uploading piece. err: <nil>
Dec 28 15:50:25.643572447 (552h21m25.514655503s) [noise=true , address=145.79.227.32:4834 , piece_id=RQ6RGHKAI5LKKPXZ6JYHIJH23GWMTWUOOI7WIPHNMESO63GYT6LA, seg_pos={0 0}, upload=484278]: Done uploading piece. err: <nil>
Dec 28 15:50:25.909540465 (552h21m25.7806235s) [noise=true , address=77.90.31.10:4518 , piece_id=7IPD4PRMZXZ26LHAFV3XHPNMG2EHLZ5DLIOIAXFLE7TKIEP3HLCA, seg_pos={0 0}, upload=484278]: Done uploading piece. err: <nil>
Dec 28 15:50:25.910168000 (552h21m25.781251045s) [noise=true , address=77.90.31.17:4770 , piece_id=5AAADVJS2KFHLTX2U7XWRFG7GZUT6JRC43T6NEE3KCQOSCZHJBYQ, seg_pos={0 0}, upload=484278]: Done uploading piece. err: <nil>
Dec 28 15:50:25.965625260 (552h21m25.836708305s) [noise=true , address=45.91.116.4:4940 , piece_id=IDCU4MCBZURJU7GA7N2HYYSCANAA2FHV44RVUX2FSG7EIIXWEF5Q, seg_pos={0 0}, upload=484278]: Done uploading piece. err: <nil>
Dec 28 15:50:25.965653274 (552h21m25.836736309s) [seg_pos={0 0}, upload=484278]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:50:25.966179005 (552h21m25.83726205s) [noise=true , address=45.91.116.8:4045 , piece_id=MUMTG4I7R3U3OGCQGEO5N2YDSIWDWVR6WDBBHL3ATSSEQOEDMREA, seg_pos={0 0}, upload=484278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:49570->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:49570->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49570->45.91.116.8:4045: use of closed network connection
Dec 28 15:50:25.966199234 (552h21m25.837282269s) [noise=true , address=45.91.116.8:4045 , piece_id=MUMTG4I7R3U3OGCQGEO5N2YDSIWDWVR6WDBBHL3ATSSEQOEDMREA, seg_pos={0 0}, upload=484278]: Long tail cancellation due to optimal threshold
Dec 28 15:50:25.966408382 (552h21m25.837491417s) [noise=true , address=45.91.116.11:4120 , piece_id=TQGAJ6MQA4GGNIZ33N56JMR4XN3GNOIZLKPV6XEX4K5EPVXS5D2A, seg_pos={0 0}, upload=484278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 28 15:50:25.966419813 (552h21m25.837502868s) [noise=true , address=45.91.116.11:4120 , piece_id=TQGAJ6MQA4GGNIZ33N56JMR4XN3GNOIZLKPV6XEX4K5EPVXS5D2A, seg_pos={0 0}, upload=484278]: Long tail cancellation due to optimal threshold
Dec 28 15:50:25.966517519 (552h21m25.837600575s) [noise=true , address=45.91.116.12:4226 , piece_id=VVYSNSK2C25K7WW2274OE5CUWCVD4HCIL6BROEKVVZG2AEC3U7PQ, seg_pos={0 0}, upload=484278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:60756->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:60756->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60756->45.91.116.12:4226: use of closed network connection
Dec 28 15:50:25.966530654 (552h21m25.837613719s) [noise=true , address=45.91.116.12:4226 , piece_id=VVYSNSK2C25K7WW2274OE5CUWCVD4HCIL6BROEKVVZG2AEC3U7PQ, seg_pos={0 0}, upload=484278]: Long tail cancellation due to optimal threshold
Dec 28 15:50:25.966560221 (552h21m25.837643266s) [seg_pos={0 0}, upload=484278]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:50:25.966696901 (552h21m25.837779946s) [upload=484278]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:50:25.966704074 (552h21m25.837787119s) [upload=484278]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:50:26.259494968 (552h21m26.130578003s) [upload=484278]: Done uploading segments...
Dec 28 15:50:26.259520175 (552h21m26.13060322s) [upload=484278]: Upload finished. err: <nil>
Duration: 30.003070023s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
12.893107612s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 15:50:37 UTC)
orbiter warning: context deadline exceeded
Duration: 12.893133381s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..............X....X...X........................X...........] duration:
30.691904698s trace:
https://jaeger.dev.storj.io/trace/3cc795a000779b86
FAIL (2025-12-28 15:50:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ) context deadline exceeded; piecestore: (Node ID: 128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, Piece ID: GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q) context deadline exceeded; piecestore: (Node ID: 12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, Piece ID: TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A) context deadline exceeded
======LOGS======
48.156469147 (552h11m48.027552193s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157156416 (552h11m48.028239461s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157556748 (552h11m48.028639793s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157945468 (552h11m48.029028503s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158645151 (552h11m48.029728196s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158842136 (552h11m48.029925171s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158983275 (552h11m48.03006631s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159344152 (552h11m48.030427187s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159350725 (552h11m48.03043376s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159710760 (552h11m48.030793795s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159732391 (552h11m48.030815437s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160068310 (552h11m48.031151346s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160439438 (552h11m48.031522473s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160813179 (552h11m48.031896225s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160839249 (552h11m48.031922284s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161271813 (552h11m48.032354848s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161765813 (552h11m48.032848848s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:49.456221594 (552h11m49.327304629s) [upload=484135]: Next returned nil segment
Dec 28 15:40:49.456235460 (552h11m49.327318495s) [upload=484135]: Waiting for error group managing segments...
Dec 28 15:40:49.789381987 (552h11m49.660465032s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.818159755 (552h11m49.6892428s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.901730607 (552h11m49.772813642s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.297735267 (552h11m50.168818312s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.866598958 (552h11m50.737682003s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.955467169 (552h11m50.826550214s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.029562809 (552h11m50.900645854s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.033202229 (552h11m50.904285264s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.119305864 (552h11m50.990388909s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.271054029 (552h11m51.142137074s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.374827580 (552h11m51.245910626s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.380607978 (552h11m51.251691013s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.500440018 (552h11m51.371523063s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688250240 (552h11m51.559333285s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688275438 (552h11m51.559358473s) [seg_pos={0 0}, upload=484135]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:40:51.688501768 (552h11m51.559584804s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688512109 (552h11m51.559595144s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688551694 (552h11m51.559634729s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688560741 (552h11m51.559643786s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688768767 (552h11m51.559851802s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWqAGD7Vrj2yzEVyNbfPp278Gga42xPLyAcdzGvUSuz3EgJYh): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688776031 (552h11m51.559859066s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688811729 (552h11m51.559894784s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688820707 (552h11m51.559903752s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688840654 (552h11m51.559923689s) [seg_pos={0 0}, upload=484135]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:40:51.688958800 (552h11m51.560041845s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:40:51.688965582 (552h11m51.560048617s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:40:51.990040599 (552h11m51.861123664s) [upload=484135]: Done uploading segments...
Dec 28 15:40:51.990091757 (552h11m51.861174802s) [upload=484135]: Upload finished. err: <nil>
Duration: 30.691934745s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......X................................X...................] duration:
16.726773914s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:50:24 UTC)
orbiter warning: context deadline exceeded
Duration: 16.726852785s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
38 out of 60 is failed (threshold=6) [.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX..XX...XXX.....XX.] duration:
3.518505466s trace:
https://jaeger.dev.storj.io/trace/721ddd16157f1400
FAIL (2025-12-28 15:50:22 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.518531485s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [.X.....X......XX.X..XXX.X.XXX.XX.XX...X.X........X.X.X.X....] duration:
9.763145487s trace:
https://jaeger.dev.storj.io/trace/6fd306ab031b44ee
FAIL (2025-12-28 15:50:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: LZOAI6GWSLU4HVHMUNAUWD7XD4OWRW3TMM6LV3BFMCIQRBOUCLXQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: T5DZHFA3K4QZI55NTSUWXEQEE4EFAP2AM2CHWO5BXZC5V47QUOLQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: VXT5RIWNO4YY5PPXECL24J3Y2JIJGYBPMFE3TLOXHPRF5PB57JBQ) context deadline exceeded
======LOGS======
Dec 28 15:46:42.551758296 (552h17m42.422841342s) [upload=484225]: Starting upload
Dec 28 15:46:42.551915806 (552h17m42.422998851s) [upload=484225]: Done starting upload
Dec 28 15:46:42.552009895 (552h17m42.42309293s) [upload=484225]: Uploading segments...
Dec 28 15:46:42.552240514 (552h17m42.423323559s) [upload=484225]: Got next segment. Inline: false
Dec 28 15:46:42.552306449 (552h17m42.423389495s) [upload=484225]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:46:42.552315588 (552h17m42.423398633s) [upload=484225]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:46:42.816464651 (552h17m42.687547696s) [seg_pos={0 0}, upload=484225]: Begin upload segment...
Dec 28 15:46:42.816519366 (552h17m42.687602401s) [seg_pos={0 0}, upload=484225]: Done begin upload segment.
Dec 28 15:46:42.816551005 (552h17m42.68763404s) [noise=true , address=77.90.31.12:4590 , piece_id=LZOAI6GWSLU4HVHMUNAUWD7XD4OWRW3TMM6LV3BFMCIQRBOUCLXQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.816645595 (552h17m42.68772863s) [noise=true , address=145.79.227.42:4122 , piece_id=VMFRYTCHGHPCWX7VVASLZNOYO42MF6GKIBK27Z6KHXHW6SRGYFSQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.817258433 (552h17m42.688341478s) [noise=true , address=145.79.227.19:4078 , piece_id=UXZJGEGBHGENUT65WA3ASHMDAYFTUNXSDDL6SLGOZCTNK7GXNR6A, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.817645119 (552h17m42.688728154s) [noise=true , address=77.90.31.10:4520 , piece_id=T5DZHFA3K4QZI55NTSUWXEQEE4EFAP2AM2CHWO5BXZC5V47QUOLQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.817889093 (552h17m42.688972128s) [noise=true , address=77.90.31.13:4629 , piece_id=VXT5RIWNO4YY5PPXECL24J3Y2JIJGYBPMFE3TLOXHPRF5PB57JBQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818067152 (552h17m42.689150187s) [noise=true , address=145.79.227.6:4259 , piece_id=VLDEVIATMLTPW7C4YMDKUR5DLXOXXMERLCWE4RN6HDERHXJ53VVA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818338649 (552h17m42.689421685s) [noise=true , address=145.79.227.7:4295 , piece_id=SY2PT47VSJ2TID4Y6XC7TCX3TCPTV5BTYV423ZJ2LZH4JVJC5ROQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818476963 (552h17m42.689559998s) [noise=true , address=77.90.31.21:4880 , piece_id=EMFCHYE3JV4ZN6HGPTS44S2XE7UCJ4XX6VECRPU7NQLYL6MQOCHA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818660542 (552h17m42.689743577s) [noise=true , address=145.79.227.27:4691 , piece_id=WYEVTHSUNDPR2EAO527L3XXGR7CJ6JKY7DMCYEDJOT3AUXKZ7WFQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818916569 (552h17m42.689999604s) [noise=true , address=145.79.227.40:4049 , piece_id=66GVXKFY5J6YE6KAFID7ETGHJIDPEK7K775AGBSONIF6XGILIMFA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.819021549 (552h17m42.690104584s) [noise=true , address=145.79.227.3:4150 , piece_id=QVDXLVQH7Z2QQDHGRPSXAKJOPQDE2GSY3ONADAFEJZCDPEP7OETQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.819163069 (552h17m42.690246114s) [noise=true , address=77.90.31.19:4844 , piece_id=BQPHVJES2H6GAVKFMU6EPJSJHCXIVAFOJ5AFA6FUTFOR2DUMNPOA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.819627434 (552h17m42.690710469s) [noise=true , address=145.79.227.15:4875 , piece_id=N4XV5SACDNTTF7KXMRQUOF77LL7HFXWCJIL5YZFLBPYT5CPQF2RQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.820097719 (552h17m42.691180764s) [noise=true , address=145.79.227.41:4086 , piece_id=5XSLZQVEEFWB6ZEDVZZQMI2Q2KESZCEOFIRK6YEIGAJCJY3H7WGA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.820517368 (552h17m42.691600413s) [noise=true , address=145.79.227.2:4114 , piece_id=J7FC5V2NNJC267AYTTP2BR3DJE3U7UGFOEPBO2VNNO3AFSIZ3C7A, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.821041806 (552h17m42.692124851s) [noise=true , address=145.79.227.49:4339 , piece_id=HTQE56GE6N6KBML2FVFXM3JE7Q6FO7FKESJIC6JTE5KHBQB7KU7Q, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:44.499663946 (552h17m44.370746991s) [upload=484225]: Next returned nil segment
Dec 28 15:46:44.499678063 (552h17m44.370761098s) [upload=484225]: Waiting for error group managing segments...
Dec 28 15:46:44.821148457 (552h17m44.692231502s) [noise=true , address=145.79.227.6:4259 , piece_id=VLDEVIATMLTPW7C4YMDKUR5DLXOXXMERLCWE4RN6HDERHXJ53VVA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.825878193 (552h17m44.696961228s) [noise=true , address=145.79.227.27:4691 , piece_id=WYEVTHSUNDPR2EAO527L3XXGR7CJ6JKY7DMCYEDJOT3AUXKZ7WFQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.826120755 (552h17m44.69720379s) [noise=true , address=145.79.227.19:4078 , piece_id=UXZJGEGBHGENUT65WA3ASHMDAYFTUNXSDDL6SLGOZCTNK7GXNR6A, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.829707705 (552h17m44.70079075s) [noise=true , address=145.79.227.41:4086 , piece_id=5XSLZQVEEFWB6ZEDVZZQMI2Q2KESZCEOFIRK6YEIGAJCJY3H7WGA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.836299617 (552h17m44.707382652s) [noise=true , address=145.79.227.15:4875 , piece_id=N4XV5SACDNTTF7KXMRQUOF77LL7HFXWCJIL5YZFLBPYT5CPQF2RQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.836585912 (552h17m44.707668957s) [noise=true , address=145.79.227.7:4295 , piece_id=SY2PT47VSJ2TID4Y6XC7TCX3TCPTV5BTYV423ZJ2LZH4JVJC5ROQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.846605191 (552h17m44.717688226s) [noise=true , address=145.79.227.42:4122 , piece_id=VMFRYTCHGHPCWX7VVASLZNOYO42MF6GKIBK27Z6KHXHW6SRGYFSQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.849890456 (552h17m44.720973491s) [noise=true , address=145.79.227.3:4150 , piece_id=QVDXLVQH7Z2QQDHGRPSXAKJOPQDE2GSY3ONADAFEJZCDPEP7OETQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.853396733 (552h17m44.724479768s) [noise=true , address=145.79.227.2:4114 , piece_id=J7FC5V2NNJC267AYTTP2BR3DJE3U7UGFOEPBO2VNNO3AFSIZ3C7A, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.989310715 (552h17m44.86039376s) [noise=true , address=77.90.31.12:4590 , piece_id=LZOAI6GWSLU4HVHMUNAUWD7XD4OWRW3TMM6LV3BFMCIQRBOUCLXQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:45.109910548 (552h17m44.980993583s) [noise=true , address=77.90.31.10:4520 , piece_id=T5DZHFA3K4QZI55NTSUWXEQEE4EFAP2AM2CHWO5BXZC5V47QUOLQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:45.182016379 (552h17m45.053099414s) [noise=true , address=77.90.31.13:4629 , piece_id=VXT5RIWNO4YY5PPXECL24J3Y2JIJGYBPMFE3TLOXHPRF5PB57JBQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:45.725747105 (552h17m45.59683014s) [noise=true , address=77.90.31.21:4880 , piece_id=EMFCHYE3JV4ZN6HGPTS44S2XE7UCJ4XX6VECRPU7NQLYL6MQOCHA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:45.725765800 (552h17m45.596848835s) [seg_pos={0 0}, upload=484225]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:46:45.726011879 (552h17m45.597094914s) [noise=true , address=145.79.227.40:4049 , piece_id=66GVXKFY5J6YE6KAFID7ETGHJIDPEK7K775AGBSONIF6XGILIMFA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 28 15:46:45.726021828 (552h17m45.597104863s) [noise=true , address=145.79.227.40:4049 , piece_id=66GVXKFY5J6YE6KAFID7ETGHJIDPEK7K775AGBSONIF6XGILIMFA, seg_pos={0 0}, upload=484225]: Long tail cancellation due to optimal threshold
Dec 28 15:46:45.726109043 (552h17m45.597192079s) [noise=true , address=77.90.31.19:4844 , piece_id=BQPHVJES2H6GAVKFMU6EPJSJHCXIVAFOJ5AFA6FUTFOR2DUMNPOA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 28 15:46:45.726125305 (552h17m45.59720834s) [noise=true , address=77.90.31.19:4844 , piece_id=BQPHVJES2H6GAVKFMU6EPJSJHCXIVAFOJ5AFA6FUTFOR2DUMNPOA, seg_pos={0 0}, upload=484225]: Long tail cancellation due to optimal threshold
Dec 28 15:46:45.726145483 (552h17m45.597228518s) [noise=true , address=145.79.227.49:4339 , piece_id=HTQE56GE6N6KBML2FVFXM3JE7Q6FO7FKESJIC6JTE5KHBQB7KU7Q, seg_pos={0 0}, upload=484225]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 28 15:46:45.726151856 (552h17m45.597234891s) [noise=true , address=145.79.227.49:4339 , piece_id=HTQE56GE6N6KBML2FVFXM3JE7Q6FO7FKESJIC6JTE5KHBQB7KU7Q, seg_pos={0 0}, upload=484225]: Long tail cancellation due to optimal threshold
Dec 28 15:46:45.726170340 (552h17m45.597253375s) [seg_pos={0 0}, upload=484225]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:46:45.726236677 (552h17m45.597319712s) [upload=484225]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:46:45.726241606 (552h17m45.597324641s) [upload=484225]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:46:46.016393204 (552h17m45.887476249s) [upload=484225]: Done uploading segments...
Dec 28 15:46:46.016418120 (552h17m45.887501166s) [upload=484225]: Upload finished. err: <nil>
Duration: 9.763801638s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
39 out of 60 is failed (threshold=6) [X.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX..XX...XXX.....XX] duration:
5.001189755s trace:
https://jaeger.dev.storj.io/trace/721ddd16157f1400
FAIL (2025-12-28 15:49:59 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001254478s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........X................................X..................] duration:
11.912585863s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:49:41 UTC)
orbiter warning: context deadline exceeded
Duration: 11.912615158s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [X.X.....X......XX.X..XXX.X.XXX.XX.XX...X.X........X.X.X.X...] duration:
6.045840938s trace:
https://jaeger.dev.storj.io/trace/6fd306ab031b44ee
FAIL (2025-12-28 15:49:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: LZOAI6GWSLU4HVHMUNAUWD7XD4OWRW3TMM6LV3BFMCIQRBOUCLXQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: T5DZHFA3K4QZI55NTSUWXEQEE4EFAP2AM2CHWO5BXZC5V47QUOLQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: VXT5RIWNO4YY5PPXECL24J3Y2JIJGYBPMFE3TLOXHPRF5PB57JBQ) context deadline exceeded
======LOGS======
Dec 28 15:46:42.551758296 (552h17m42.422841342s) [upload=484225]: Starting upload
Dec 28 15:46:42.551915806 (552h17m42.422998851s) [upload=484225]: Done starting upload
Dec 28 15:46:42.552009895 (552h17m42.42309293s) [upload=484225]: Uploading segments...
Dec 28 15:46:42.552240514 (552h17m42.423323559s) [upload=484225]: Got next segment. Inline: false
Dec 28 15:46:42.552306449 (552h17m42.423389495s) [upload=484225]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:46:42.552315588 (552h17m42.423398633s) [upload=484225]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:46:42.816464651 (552h17m42.687547696s) [seg_pos={0 0}, upload=484225]: Begin upload segment...
Dec 28 15:46:42.816519366 (552h17m42.687602401s) [seg_pos={0 0}, upload=484225]: Done begin upload segment.
Dec 28 15:46:42.816551005 (552h17m42.68763404s) [noise=true , address=77.90.31.12:4590 , piece_id=LZOAI6GWSLU4HVHMUNAUWD7XD4OWRW3TMM6LV3BFMCIQRBOUCLXQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.816645595 (552h17m42.68772863s) [noise=true , address=145.79.227.42:4122 , piece_id=VMFRYTCHGHPCWX7VVASLZNOYO42MF6GKIBK27Z6KHXHW6SRGYFSQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.817258433 (552h17m42.688341478s) [noise=true , address=145.79.227.19:4078 , piece_id=UXZJGEGBHGENUT65WA3ASHMDAYFTUNXSDDL6SLGOZCTNK7GXNR6A, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.817645119 (552h17m42.688728154s) [noise=true , address=77.90.31.10:4520 , piece_id=T5DZHFA3K4QZI55NTSUWXEQEE4EFAP2AM2CHWO5BXZC5V47QUOLQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.817889093 (552h17m42.688972128s) [noise=true , address=77.90.31.13:4629 , piece_id=VXT5RIWNO4YY5PPXECL24J3Y2JIJGYBPMFE3TLOXHPRF5PB57JBQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818067152 (552h17m42.689150187s) [noise=true , address=145.79.227.6:4259 , piece_id=VLDEVIATMLTPW7C4YMDKUR5DLXOXXMERLCWE4RN6HDERHXJ53VVA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818338649 (552h17m42.689421685s) [noise=true , address=145.79.227.7:4295 , piece_id=SY2PT47VSJ2TID4Y6XC7TCX3TCPTV5BTYV423ZJ2LZH4JVJC5ROQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818476963 (552h17m42.689559998s) [noise=true , address=77.90.31.21:4880 , piece_id=EMFCHYE3JV4ZN6HGPTS44S2XE7UCJ4XX6VECRPU7NQLYL6MQOCHA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818660542 (552h17m42.689743577s) [noise=true , address=145.79.227.27:4691 , piece_id=WYEVTHSUNDPR2EAO527L3XXGR7CJ6JKY7DMCYEDJOT3AUXKZ7WFQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818916569 (552h17m42.689999604s) [noise=true , address=145.79.227.40:4049 , piece_id=66GVXKFY5J6YE6KAFID7ETGHJIDPEK7K775AGBSONIF6XGILIMFA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.819021549 (552h17m42.690104584s) [noise=true , address=145.79.227.3:4150 , piece_id=QVDXLVQH7Z2QQDHGRPSXAKJOPQDE2GSY3ONADAFEJZCDPEP7OETQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.819163069 (552h17m42.690246114s) [noise=true , address=77.90.31.19:4844 , piece_id=BQPHVJES2H6GAVKFMU6EPJSJHCXIVAFOJ5AFA6FUTFOR2DUMNPOA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.819627434 (552h17m42.690710469s) [noise=true , address=145.79.227.15:4875 , piece_id=N4XV5SACDNTTF7KXMRQUOF77LL7HFXWCJIL5YZFLBPYT5CPQF2RQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.820097719 (552h17m42.691180764s) [noise=true , address=145.79.227.41:4086 , piece_id=5XSLZQVEEFWB6ZEDVZZQMI2Q2KESZCEOFIRK6YEIGAJCJY3H7WGA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.820517368 (552h17m42.691600413s) [noise=true , address=145.79.227.2:4114 , piece_id=J7FC5V2NNJC267AYTTP2BR3DJE3U7UGFOEPBO2VNNO3AFSIZ3C7A, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.821041806 (552h17m42.692124851s) [noise=true , address=145.79.227.49:4339 , piece_id=HTQE56GE6N6KBML2FVFXM3JE7Q6FO7FKESJIC6JTE5KHBQB7KU7Q, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:44.499663946 (552h17m44.370746991s) [upload=484225]: Next returned nil segment
Dec 28 15:46:44.499678063 (552h17m44.370761098s) [upload=484225]: Waiting for error group managing segments...
Dec 28 15:46:44.821148457 (552h17m44.692231502s) [noise=true , address=145.79.227.6:4259 , piece_id=VLDEVIATMLTPW7C4YMDKUR5DLXOXXMERLCWE4RN6HDERHXJ53VVA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.825878193 (552h17m44.696961228s) [noise=true , address=145.79.227.27:4691 , piece_id=WYEVTHSUNDPR2EAO527L3XXGR7CJ6JKY7DMCYEDJOT3AUXKZ7WFQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.826120755 (552h17m44.69720379s) [noise=true , address=145.79.227.19:4078 , piece_id=UXZJGEGBHGENUT65WA3ASHMDAYFTUNXSDDL6SLGOZCTNK7GXNR6A, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.829707705 (552h17m44.70079075s) [noise=true , address=145.79.227.41:4086 , piece_id=5XSLZQVEEFWB6ZEDVZZQMI2Q2KESZCEOFIRK6YEIGAJCJY3H7WGA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.836299617 (552h17m44.707382652s) [noise=true , address=145.79.227.15:4875 , piece_id=N4XV5SACDNTTF7KXMRQUOF77LL7HFXWCJIL5YZFLBPYT5CPQF2RQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.836585912 (552h17m44.707668957s) [noise=true , address=145.79.227.7:4295 , piece_id=SY2PT47VSJ2TID4Y6XC7TCX3TCPTV5BTYV423ZJ2LZH4JVJC5ROQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.846605191 (552h17m44.717688226s) [noise=true , address=145.79.227.42:4122 , piece_id=VMFRYTCHGHPCWX7VVASLZNOYO42MF6GKIBK27Z6KHXHW6SRGYFSQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.849890456 (552h17m44.720973491s) [noise=true , address=145.79.227.3:4150 , piece_id=QVDXLVQH7Z2QQDHGRPSXAKJOPQDE2GSY3ONADAFEJZCDPEP7OETQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.853396733 (552h17m44.724479768s) [noise=true , address=145.79.227.2:4114 , piece_id=J7FC5V2NNJC267AYTTP2BR3DJE3U7UGFOEPBO2VNNO3AFSIZ3C7A, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.989310715 (552h17m44.86039376s) [noise=true , address=77.90.31.12:4590 , piece_id=LZOAI6GWSLU4HVHMUNAUWD7XD4OWRW3TMM6LV3BFMCIQRBOUCLXQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:45.109910548 (552h17m44.980993583s) [noise=true , address=77.90.31.10:4520 , piece_id=T5DZHFA3K4QZI55NTSUWXEQEE4EFAP2AM2CHWO5BXZC5V47QUOLQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:45.182016379 (552h17m45.053099414s) [noise=true , address=77.90.31.13:4629 , piece_id=VXT5RIWNO4YY5PPXECL24J3Y2JIJGYBPMFE3TLOXHPRF5PB57JBQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:45.725747105 (552h17m45.59683014s) [noise=true , address=77.90.31.21:4880 , piece_id=EMFCHYE3JV4ZN6HGPTS44S2XE7UCJ4XX6VECRPU7NQLYL6MQOCHA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:45.725765800 (552h17m45.596848835s) [seg_pos={0 0}, upload=484225]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:46:45.726011879 (552h17m45.597094914s) [noise=true , address=145.79.227.40:4049 , piece_id=66GVXKFY5J6YE6KAFID7ETGHJIDPEK7K775AGBSONIF6XGILIMFA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 28 15:46:45.726021828 (552h17m45.597104863s) [noise=true , address=145.79.227.40:4049 , piece_id=66GVXKFY5J6YE6KAFID7ETGHJIDPEK7K775AGBSONIF6XGILIMFA, seg_pos={0 0}, upload=484225]: Long tail cancellation due to optimal threshold
Dec 28 15:46:45.726109043 (552h17m45.597192079s) [noise=true , address=77.90.31.19:4844 , piece_id=BQPHVJES2H6GAVKFMU6EPJSJHCXIVAFOJ5AFA6FUTFOR2DUMNPOA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 28 15:46:45.726125305 (552h17m45.59720834s) [noise=true , address=77.90.31.19:4844 , piece_id=BQPHVJES2H6GAVKFMU6EPJSJHCXIVAFOJ5AFA6FUTFOR2DUMNPOA, seg_pos={0 0}, upload=484225]: Long tail cancellation due to optimal threshold
Dec 28 15:46:45.726145483 (552h17m45.597228518s) [noise=true , address=145.79.227.49:4339 , piece_id=HTQE56GE6N6KBML2FVFXM3JE7Q6FO7FKESJIC6JTE5KHBQB7KU7Q, seg_pos={0 0}, upload=484225]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 28 15:46:45.726151856 (552h17m45.597234891s) [noise=true , address=145.79.227.49:4339 , piece_id=HTQE56GE6N6KBML2FVFXM3JE7Q6FO7FKESJIC6JTE5KHBQB7KU7Q, seg_pos={0 0}, upload=484225]: Long tail cancellation due to optimal threshold
Dec 28 15:46:45.726170340 (552h17m45.597253375s) [seg_pos={0 0}, upload=484225]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:46:45.726236677 (552h17m45.597319712s) [upload=484225]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:46:45.726241606 (552h17m45.597324641s) [upload=484225]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:46:46.016393204 (552h17m45.887476249s) [upload=484225]: Done uploading segments...
Dec 28 15:46:46.016418120 (552h17m45.887501166s) [upload=484225]: Upload finished. err: <nil>
Duration: 6.045870002s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
13.864921707s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 15:49:35 UTC)
orbiter warning: context deadline exceeded
Duration: 13.865051505s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
38 out of 60 is failed (threshold=6) [.X.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX..XX...XXX.....X] duration:
5.001708047s trace:
https://jaeger.dev.storj.io/trace/6b19580419447156
FAIL (2025-12-28 15:49:29 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001732304s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...............X....X...X........................X..........] duration:
37.959258952s trace:
https://jaeger.dev.storj.io/trace/3cc795a000779b86
FAIL (2025-12-28 15:49:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ) context deadline exceeded; piecestore: (Node ID: 128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, Piece ID: GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q) context deadline exceeded; piecestore: (Node ID: 12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, Piece ID: TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A) context deadline exceeded
======LOGS======
48.156469147 (552h11m48.027552193s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157156416 (552h11m48.028239461s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157556748 (552h11m48.028639793s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157945468 (552h11m48.029028503s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158645151 (552h11m48.029728196s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158842136 (552h11m48.029925171s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158983275 (552h11m48.03006631s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159344152 (552h11m48.030427187s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159350725 (552h11m48.03043376s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159710760 (552h11m48.030793795s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159732391 (552h11m48.030815437s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160068310 (552h11m48.031151346s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160439438 (552h11m48.031522473s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160813179 (552h11m48.031896225s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160839249 (552h11m48.031922284s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161271813 (552h11m48.032354848s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161765813 (552h11m48.032848848s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:49.456221594 (552h11m49.327304629s) [upload=484135]: Next returned nil segment
Dec 28 15:40:49.456235460 (552h11m49.327318495s) [upload=484135]: Waiting for error group managing segments...
Dec 28 15:40:49.789381987 (552h11m49.660465032s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.818159755 (552h11m49.6892428s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.901730607 (552h11m49.772813642s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.297735267 (552h11m50.168818312s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.866598958 (552h11m50.737682003s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.955467169 (552h11m50.826550214s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.029562809 (552h11m50.900645854s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.033202229 (552h11m50.904285264s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.119305864 (552h11m50.990388909s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.271054029 (552h11m51.142137074s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.374827580 (552h11m51.245910626s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.380607978 (552h11m51.251691013s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.500440018 (552h11m51.371523063s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688250240 (552h11m51.559333285s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688275438 (552h11m51.559358473s) [seg_pos={0 0}, upload=484135]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:40:51.688501768 (552h11m51.559584804s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688512109 (552h11m51.559595144s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688551694 (552h11m51.559634729s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688560741 (552h11m51.559643786s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688768767 (552h11m51.559851802s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWqAGD7Vrj2yzEVyNbfPp278Gga42xPLyAcdzGvUSuz3EgJYh): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688776031 (552h11m51.559859066s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688811729 (552h11m51.559894784s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688820707 (552h11m51.559903752s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688840654 (552h11m51.559923689s) [seg_pos={0 0}, upload=484135]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:40:51.688958800 (552h11m51.560041845s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:40:51.688965582 (552h11m51.560048617s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:40:51.990040599 (552h11m51.861123664s) [upload=484135]: Done uploading segments...
Dec 28 15:40:51.990091757 (552h11m51.861174802s) [upload=484135]: Upload finished. err: <nil>
Duration: 37.95927358s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........X................................X.................] duration:
13.21595829s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:49:21 UTC)
orbiter warning: context deadline exceeded
Duration: 13.215987706s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [................X....X...X........................X.........] duration:
45.720114096s trace:
https://jaeger.dev.storj.io/trace/3cc795a000779b86
FAIL (2025-12-28 15:48:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ) context deadline exceeded; piecestore: (Node ID: 128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, Piece ID: GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q) context deadline exceeded; piecestore: (Node ID: 12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, Piece ID: TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A) context deadline exceeded
======LOGS======
48.156469147 (552h11m48.027552193s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157156416 (552h11m48.028239461s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157556748 (552h11m48.028639793s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157945468 (552h11m48.029028503s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158645151 (552h11m48.029728196s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158842136 (552h11m48.029925171s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158983275 (552h11m48.03006631s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159344152 (552h11m48.030427187s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159350725 (552h11m48.03043376s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159710760 (552h11m48.030793795s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159732391 (552h11m48.030815437s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160068310 (552h11m48.031151346s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160439438 (552h11m48.031522473s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160813179 (552h11m48.031896225s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160839249 (552h11m48.031922284s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161271813 (552h11m48.032354848s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161765813 (552h11m48.032848848s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:49.456221594 (552h11m49.327304629s) [upload=484135]: Next returned nil segment
Dec 28 15:40:49.456235460 (552h11m49.327318495s) [upload=484135]: Waiting for error group managing segments...
Dec 28 15:40:49.789381987 (552h11m49.660465032s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.818159755 (552h11m49.6892428s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.901730607 (552h11m49.772813642s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.297735267 (552h11m50.168818312s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.866598958 (552h11m50.737682003s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.955467169 (552h11m50.826550214s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.029562809 (552h11m50.900645854s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.033202229 (552h11m50.904285264s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.119305864 (552h11m50.990388909s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.271054029 (552h11m51.142137074s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.374827580 (552h11m51.245910626s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.380607978 (552h11m51.251691013s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.500440018 (552h11m51.371523063s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688250240 (552h11m51.559333285s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688275438 (552h11m51.559358473s) [seg_pos={0 0}, upload=484135]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:40:51.688501768 (552h11m51.559584804s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688512109 (552h11m51.559595144s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688551694 (552h11m51.559634729s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688560741 (552h11m51.559643786s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688768767 (552h11m51.559851802s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWqAGD7Vrj2yzEVyNbfPp278Gga42xPLyAcdzGvUSuz3EgJYh): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688776031 (552h11m51.559859066s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688811729 (552h11m51.559894784s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688820707 (552h11m51.559903752s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688840654 (552h11m51.559923689s) [seg_pos={0 0}, upload=484135]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:40:51.688958800 (552h11m51.560041845s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:40:51.688965582 (552h11m51.560048617s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:40:51.990040599 (552h11m51.861123664s) [upload=484135]: Done uploading segments...
Dec 28 15:40:51.990091757 (552h11m51.861174802s) [upload=484135]: Upload finished. err: <nil>
Duration: 45.720144253s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [.X.X.....X......XX.X..XXX.X.XXX.XX.XX...X.X........X.X.X.X..] duration:
5.68450421s trace:
https://jaeger.dev.storj.io/trace/6fd306ab031b44ee
FAIL (2025-12-28 15:48:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: LZOAI6GWSLU4HVHMUNAUWD7XD4OWRW3TMM6LV3BFMCIQRBOUCLXQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: T5DZHFA3K4QZI55NTSUWXEQEE4EFAP2AM2CHWO5BXZC5V47QUOLQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: VXT5RIWNO4YY5PPXECL24J3Y2JIJGYBPMFE3TLOXHPRF5PB57JBQ) context deadline exceeded
======LOGS======
Dec 28 15:46:42.551758296 (552h17m42.422841342s) [upload=484225]: Starting upload
Dec 28 15:46:42.551915806 (552h17m42.422998851s) [upload=484225]: Done starting upload
Dec 28 15:46:42.552009895 (552h17m42.42309293s) [upload=484225]: Uploading segments...
Dec 28 15:46:42.552240514 (552h17m42.423323559s) [upload=484225]: Got next segment. Inline: false
Dec 28 15:46:42.552306449 (552h17m42.423389495s) [upload=484225]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:46:42.552315588 (552h17m42.423398633s) [upload=484225]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:46:42.816464651 (552h17m42.687547696s) [seg_pos={0 0}, upload=484225]: Begin upload segment...
Dec 28 15:46:42.816519366 (552h17m42.687602401s) [seg_pos={0 0}, upload=484225]: Done begin upload segment.
Dec 28 15:46:42.816551005 (552h17m42.68763404s) [noise=true , address=77.90.31.12:4590 , piece_id=LZOAI6GWSLU4HVHMUNAUWD7XD4OWRW3TMM6LV3BFMCIQRBOUCLXQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.816645595 (552h17m42.68772863s) [noise=true , address=145.79.227.42:4122 , piece_id=VMFRYTCHGHPCWX7VVASLZNOYO42MF6GKIBK27Z6KHXHW6SRGYFSQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.817258433 (552h17m42.688341478s) [noise=true , address=145.79.227.19:4078 , piece_id=UXZJGEGBHGENUT65WA3ASHMDAYFTUNXSDDL6SLGOZCTNK7GXNR6A, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.817645119 (552h17m42.688728154s) [noise=true , address=77.90.31.10:4520 , piece_id=T5DZHFA3K4QZI55NTSUWXEQEE4EFAP2AM2CHWO5BXZC5V47QUOLQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.817889093 (552h17m42.688972128s) [noise=true , address=77.90.31.13:4629 , piece_id=VXT5RIWNO4YY5PPXECL24J3Y2JIJGYBPMFE3TLOXHPRF5PB57JBQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818067152 (552h17m42.689150187s) [noise=true , address=145.79.227.6:4259 , piece_id=VLDEVIATMLTPW7C4YMDKUR5DLXOXXMERLCWE4RN6HDERHXJ53VVA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818338649 (552h17m42.689421685s) [noise=true , address=145.79.227.7:4295 , piece_id=SY2PT47VSJ2TID4Y6XC7TCX3TCPTV5BTYV423ZJ2LZH4JVJC5ROQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818476963 (552h17m42.689559998s) [noise=true , address=77.90.31.21:4880 , piece_id=EMFCHYE3JV4ZN6HGPTS44S2XE7UCJ4XX6VECRPU7NQLYL6MQOCHA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818660542 (552h17m42.689743577s) [noise=true , address=145.79.227.27:4691 , piece_id=WYEVTHSUNDPR2EAO527L3XXGR7CJ6JKY7DMCYEDJOT3AUXKZ7WFQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818916569 (552h17m42.689999604s) [noise=true , address=145.79.227.40:4049 , piece_id=66GVXKFY5J6YE6KAFID7ETGHJIDPEK7K775AGBSONIF6XGILIMFA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.819021549 (552h17m42.690104584s) [noise=true , address=145.79.227.3:4150 , piece_id=QVDXLVQH7Z2QQDHGRPSXAKJOPQDE2GSY3ONADAFEJZCDPEP7OETQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.819163069 (552h17m42.690246114s) [noise=true , address=77.90.31.19:4844 , piece_id=BQPHVJES2H6GAVKFMU6EPJSJHCXIVAFOJ5AFA6FUTFOR2DUMNPOA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.819627434 (552h17m42.690710469s) [noise=true , address=145.79.227.15:4875 , piece_id=N4XV5SACDNTTF7KXMRQUOF77LL7HFXWCJIL5YZFLBPYT5CPQF2RQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.820097719 (552h17m42.691180764s) [noise=true , address=145.79.227.41:4086 , piece_id=5XSLZQVEEFWB6ZEDVZZQMI2Q2KESZCEOFIRK6YEIGAJCJY3H7WGA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.820517368 (552h17m42.691600413s) [noise=true , address=145.79.227.2:4114 , piece_id=J7FC5V2NNJC267AYTTP2BR3DJE3U7UGFOEPBO2VNNO3AFSIZ3C7A, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.821041806 (552h17m42.692124851s) [noise=true , address=145.79.227.49:4339 , piece_id=HTQE56GE6N6KBML2FVFXM3JE7Q6FO7FKESJIC6JTE5KHBQB7KU7Q, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:44.499663946 (552h17m44.370746991s) [upload=484225]: Next returned nil segment
Dec 28 15:46:44.499678063 (552h17m44.370761098s) [upload=484225]: Waiting for error group managing segments...
Dec 28 15:46:44.821148457 (552h17m44.692231502s) [noise=true , address=145.79.227.6:4259 , piece_id=VLDEVIATMLTPW7C4YMDKUR5DLXOXXMERLCWE4RN6HDERHXJ53VVA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.825878193 (552h17m44.696961228s) [noise=true , address=145.79.227.27:4691 , piece_id=WYEVTHSUNDPR2EAO527L3XXGR7CJ6JKY7DMCYEDJOT3AUXKZ7WFQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.826120755 (552h17m44.69720379s) [noise=true , address=145.79.227.19:4078 , piece_id=UXZJGEGBHGENUT65WA3ASHMDAYFTUNXSDDL6SLGOZCTNK7GXNR6A, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.829707705 (552h17m44.70079075s) [noise=true , address=145.79.227.41:4086 , piece_id=5XSLZQVEEFWB6ZEDVZZQMI2Q2KESZCEOFIRK6YEIGAJCJY3H7WGA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.836299617 (552h17m44.707382652s) [noise=true , address=145.79.227.15:4875 , piece_id=N4XV5SACDNTTF7KXMRQUOF77LL7HFXWCJIL5YZFLBPYT5CPQF2RQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.836585912 (552h17m44.707668957s) [noise=true , address=145.79.227.7:4295 , piece_id=SY2PT47VSJ2TID4Y6XC7TCX3TCPTV5BTYV423ZJ2LZH4JVJC5ROQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.846605191 (552h17m44.717688226s) [noise=true , address=145.79.227.42:4122 , piece_id=VMFRYTCHGHPCWX7VVASLZNOYO42MF6GKIBK27Z6KHXHW6SRGYFSQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.849890456 (552h17m44.720973491s) [noise=true , address=145.79.227.3:4150 , piece_id=QVDXLVQH7Z2QQDHGRPSXAKJOPQDE2GSY3ONADAFEJZCDPEP7OETQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.853396733 (552h17m44.724479768s) [noise=true , address=145.79.227.2:4114 , piece_id=J7FC5V2NNJC267AYTTP2BR3DJE3U7UGFOEPBO2VNNO3AFSIZ3C7A, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.989310715 (552h17m44.86039376s) [noise=true , address=77.90.31.12:4590 , piece_id=LZOAI6GWSLU4HVHMUNAUWD7XD4OWRW3TMM6LV3BFMCIQRBOUCLXQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:45.109910548 (552h17m44.980993583s) [noise=true , address=77.90.31.10:4520 , piece_id=T5DZHFA3K4QZI55NTSUWXEQEE4EFAP2AM2CHWO5BXZC5V47QUOLQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:45.182016379 (552h17m45.053099414s) [noise=true , address=77.90.31.13:4629 , piece_id=VXT5RIWNO4YY5PPXECL24J3Y2JIJGYBPMFE3TLOXHPRF5PB57JBQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:45.725747105 (552h17m45.59683014s) [noise=true , address=77.90.31.21:4880 , piece_id=EMFCHYE3JV4ZN6HGPTS44S2XE7UCJ4XX6VECRPU7NQLYL6MQOCHA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:45.725765800 (552h17m45.596848835s) [seg_pos={0 0}, upload=484225]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:46:45.726011879 (552h17m45.597094914s) [noise=true , address=145.79.227.40:4049 , piece_id=66GVXKFY5J6YE6KAFID7ETGHJIDPEK7K775AGBSONIF6XGILIMFA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 28 15:46:45.726021828 (552h17m45.597104863s) [noise=true , address=145.79.227.40:4049 , piece_id=66GVXKFY5J6YE6KAFID7ETGHJIDPEK7K775AGBSONIF6XGILIMFA, seg_pos={0 0}, upload=484225]: Long tail cancellation due to optimal threshold
Dec 28 15:46:45.726109043 (552h17m45.597192079s) [noise=true , address=77.90.31.19:4844 , piece_id=BQPHVJES2H6GAVKFMU6EPJSJHCXIVAFOJ5AFA6FUTFOR2DUMNPOA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 28 15:46:45.726125305 (552h17m45.59720834s) [noise=true , address=77.90.31.19:4844 , piece_id=BQPHVJES2H6GAVKFMU6EPJSJHCXIVAFOJ5AFA6FUTFOR2DUMNPOA, seg_pos={0 0}, upload=484225]: Long tail cancellation due to optimal threshold
Dec 28 15:46:45.726145483 (552h17m45.597228518s) [noise=true , address=145.79.227.49:4339 , piece_id=HTQE56GE6N6KBML2FVFXM3JE7Q6FO7FKESJIC6JTE5KHBQB7KU7Q, seg_pos={0 0}, upload=484225]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 28 15:46:45.726151856 (552h17m45.597234891s) [noise=true , address=145.79.227.49:4339 , piece_id=HTQE56GE6N6KBML2FVFXM3JE7Q6FO7FKESJIC6JTE5KHBQB7KU7Q, seg_pos={0 0}, upload=484225]: Long tail cancellation due to optimal threshold
Dec 28 15:46:45.726170340 (552h17m45.597253375s) [seg_pos={0 0}, upload=484225]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:46:45.726236677 (552h17m45.597319712s) [upload=484225]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:46:45.726241606 (552h17m45.597324641s) [upload=484225]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:46:46.016393204 (552h17m45.887476249s) [upload=484225]: Done uploading segments...
Dec 28 15:46:46.016418120 (552h17m45.887501166s) [upload=484225]: Upload finished. err: <nil>
Duration: 5.684532474s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
16.862641742s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 15:48:27 UTC)
orbiter warning: context deadline exceeded
Duration: 16.862667772s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
38 out of 60 is failed (threshold=6) [X.X.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX..XX...XXX.....] duration:
3.488053469s trace:
https://jaeger.dev.storj.io/trace/64d166e63eaa7bab
FAIL (2025-12-28 15:48:18 UTC)
orbiter warning: context deadline exceeded
Duration: 3.48807481s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........X................................X................] duration:
7.47154233s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:48:16 UTC)
orbiter warning: context deadline exceeded
Duration: 7.471566926s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.................X....X...X........................X........] duration:
32.713604213s trace:
https://jaeger.dev.storj.io/trace/3cc795a000779b86
FAIL (2025-12-28 15:48:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ) context deadline exceeded; piecestore: (Node ID: 128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, Piece ID: GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q) context deadline exceeded; piecestore: (Node ID: 12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, Piece ID: TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A) context deadline exceeded
======LOGS======
48.156469147 (552h11m48.027552193s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157156416 (552h11m48.028239461s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157556748 (552h11m48.028639793s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157945468 (552h11m48.029028503s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158645151 (552h11m48.029728196s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158842136 (552h11m48.029925171s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158983275 (552h11m48.03006631s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159344152 (552h11m48.030427187s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159350725 (552h11m48.03043376s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159710760 (552h11m48.030793795s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159732391 (552h11m48.030815437s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160068310 (552h11m48.031151346s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160439438 (552h11m48.031522473s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160813179 (552h11m48.031896225s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160839249 (552h11m48.031922284s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161271813 (552h11m48.032354848s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161765813 (552h11m48.032848848s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:49.456221594 (552h11m49.327304629s) [upload=484135]: Next returned nil segment
Dec 28 15:40:49.456235460 (552h11m49.327318495s) [upload=484135]: Waiting for error group managing segments...
Dec 28 15:40:49.789381987 (552h11m49.660465032s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.818159755 (552h11m49.6892428s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.901730607 (552h11m49.772813642s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.297735267 (552h11m50.168818312s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.866598958 (552h11m50.737682003s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.955467169 (552h11m50.826550214s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.029562809 (552h11m50.900645854s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.033202229 (552h11m50.904285264s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.119305864 (552h11m50.990388909s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.271054029 (552h11m51.142137074s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.374827580 (552h11m51.245910626s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.380607978 (552h11m51.251691013s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.500440018 (552h11m51.371523063s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688250240 (552h11m51.559333285s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688275438 (552h11m51.559358473s) [seg_pos={0 0}, upload=484135]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:40:51.688501768 (552h11m51.559584804s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688512109 (552h11m51.559595144s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688551694 (552h11m51.559634729s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688560741 (552h11m51.559643786s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688768767 (552h11m51.559851802s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWqAGD7Vrj2yzEVyNbfPp278Gga42xPLyAcdzGvUSuz3EgJYh): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688776031 (552h11m51.559859066s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688811729 (552h11m51.559894784s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688820707 (552h11m51.559903752s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688840654 (552h11m51.559923689s) [seg_pos={0 0}, upload=484135]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:40:51.688958800 (552h11m51.560041845s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:40:51.688965582 (552h11m51.560048617s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:40:51.990040599 (552h11m51.861123664s) [upload=484135]: Done uploading segments...
Dec 28 15:40:51.990091757 (552h11m51.861174802s) [upload=484135]: Upload finished. err: <nil>
Duration: 32.713844612s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........X................................X...............] duration:
14.759327557s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:47:49 UTC)
orbiter warning: context deadline exceeded
Duration: 14.759359218s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [X.X.X.....X......XX.X..XXX.X.XXX.XX.XX...X.X........X.X.X.X.] duration:
5.906024682s trace:
https://jaeger.dev.storj.io/trace/6fd306ab031b44ee
FAIL (2025-12-28 15:47:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: LZOAI6GWSLU4HVHMUNAUWD7XD4OWRW3TMM6LV3BFMCIQRBOUCLXQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: T5DZHFA3K4QZI55NTSUWXEQEE4EFAP2AM2CHWO5BXZC5V47QUOLQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: VXT5RIWNO4YY5PPXECL24J3Y2JIJGYBPMFE3TLOXHPRF5PB57JBQ) context deadline exceeded
======LOGS======
Dec 28 15:46:42.551758296 (552h17m42.422841342s) [upload=484225]: Starting upload
Dec 28 15:46:42.551915806 (552h17m42.422998851s) [upload=484225]: Done starting upload
Dec 28 15:46:42.552009895 (552h17m42.42309293s) [upload=484225]: Uploading segments...
Dec 28 15:46:42.552240514 (552h17m42.423323559s) [upload=484225]: Got next segment. Inline: false
Dec 28 15:46:42.552306449 (552h17m42.423389495s) [upload=484225]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:46:42.552315588 (552h17m42.423398633s) [upload=484225]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:46:42.816464651 (552h17m42.687547696s) [seg_pos={0 0}, upload=484225]: Begin upload segment...
Dec 28 15:46:42.816519366 (552h17m42.687602401s) [seg_pos={0 0}, upload=484225]: Done begin upload segment.
Dec 28 15:46:42.816551005 (552h17m42.68763404s) [noise=true , address=77.90.31.12:4590 , piece_id=LZOAI6GWSLU4HVHMUNAUWD7XD4OWRW3TMM6LV3BFMCIQRBOUCLXQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.816645595 (552h17m42.68772863s) [noise=true , address=145.79.227.42:4122 , piece_id=VMFRYTCHGHPCWX7VVASLZNOYO42MF6GKIBK27Z6KHXHW6SRGYFSQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.817258433 (552h17m42.688341478s) [noise=true , address=145.79.227.19:4078 , piece_id=UXZJGEGBHGENUT65WA3ASHMDAYFTUNXSDDL6SLGOZCTNK7GXNR6A, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.817645119 (552h17m42.688728154s) [noise=true , address=77.90.31.10:4520 , piece_id=T5DZHFA3K4QZI55NTSUWXEQEE4EFAP2AM2CHWO5BXZC5V47QUOLQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.817889093 (552h17m42.688972128s) [noise=true , address=77.90.31.13:4629 , piece_id=VXT5RIWNO4YY5PPXECL24J3Y2JIJGYBPMFE3TLOXHPRF5PB57JBQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818067152 (552h17m42.689150187s) [noise=true , address=145.79.227.6:4259 , piece_id=VLDEVIATMLTPW7C4YMDKUR5DLXOXXMERLCWE4RN6HDERHXJ53VVA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818338649 (552h17m42.689421685s) [noise=true , address=145.79.227.7:4295 , piece_id=SY2PT47VSJ2TID4Y6XC7TCX3TCPTV5BTYV423ZJ2LZH4JVJC5ROQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818476963 (552h17m42.689559998s) [noise=true , address=77.90.31.21:4880 , piece_id=EMFCHYE3JV4ZN6HGPTS44S2XE7UCJ4XX6VECRPU7NQLYL6MQOCHA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818660542 (552h17m42.689743577s) [noise=true , address=145.79.227.27:4691 , piece_id=WYEVTHSUNDPR2EAO527L3XXGR7CJ6JKY7DMCYEDJOT3AUXKZ7WFQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818916569 (552h17m42.689999604s) [noise=true , address=145.79.227.40:4049 , piece_id=66GVXKFY5J6YE6KAFID7ETGHJIDPEK7K775AGBSONIF6XGILIMFA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.819021549 (552h17m42.690104584s) [noise=true , address=145.79.227.3:4150 , piece_id=QVDXLVQH7Z2QQDHGRPSXAKJOPQDE2GSY3ONADAFEJZCDPEP7OETQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.819163069 (552h17m42.690246114s) [noise=true , address=77.90.31.19:4844 , piece_id=BQPHVJES2H6GAVKFMU6EPJSJHCXIVAFOJ5AFA6FUTFOR2DUMNPOA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.819627434 (552h17m42.690710469s) [noise=true , address=145.79.227.15:4875 , piece_id=N4XV5SACDNTTF7KXMRQUOF77LL7HFXWCJIL5YZFLBPYT5CPQF2RQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.820097719 (552h17m42.691180764s) [noise=true , address=145.79.227.41:4086 , piece_id=5XSLZQVEEFWB6ZEDVZZQMI2Q2KESZCEOFIRK6YEIGAJCJY3H7WGA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.820517368 (552h17m42.691600413s) [noise=true , address=145.79.227.2:4114 , piece_id=J7FC5V2NNJC267AYTTP2BR3DJE3U7UGFOEPBO2VNNO3AFSIZ3C7A, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.821041806 (552h17m42.692124851s) [noise=true , address=145.79.227.49:4339 , piece_id=HTQE56GE6N6KBML2FVFXM3JE7Q6FO7FKESJIC6JTE5KHBQB7KU7Q, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:44.499663946 (552h17m44.370746991s) [upload=484225]: Next returned nil segment
Dec 28 15:46:44.499678063 (552h17m44.370761098s) [upload=484225]: Waiting for error group managing segments...
Dec 28 15:46:44.821148457 (552h17m44.692231502s) [noise=true , address=145.79.227.6:4259 , piece_id=VLDEVIATMLTPW7C4YMDKUR5DLXOXXMERLCWE4RN6HDERHXJ53VVA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.825878193 (552h17m44.696961228s) [noise=true , address=145.79.227.27:4691 , piece_id=WYEVTHSUNDPR2EAO527L3XXGR7CJ6JKY7DMCYEDJOT3AUXKZ7WFQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.826120755 (552h17m44.69720379s) [noise=true , address=145.79.227.19:4078 , piece_id=UXZJGEGBHGENUT65WA3ASHMDAYFTUNXSDDL6SLGOZCTNK7GXNR6A, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.829707705 (552h17m44.70079075s) [noise=true , address=145.79.227.41:4086 , piece_id=5XSLZQVEEFWB6ZEDVZZQMI2Q2KESZCEOFIRK6YEIGAJCJY3H7WGA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.836299617 (552h17m44.707382652s) [noise=true , address=145.79.227.15:4875 , piece_id=N4XV5SACDNTTF7KXMRQUOF77LL7HFXWCJIL5YZFLBPYT5CPQF2RQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.836585912 (552h17m44.707668957s) [noise=true , address=145.79.227.7:4295 , piece_id=SY2PT47VSJ2TID4Y6XC7TCX3TCPTV5BTYV423ZJ2LZH4JVJC5ROQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.846605191 (552h17m44.717688226s) [noise=true , address=145.79.227.42:4122 , piece_id=VMFRYTCHGHPCWX7VVASLZNOYO42MF6GKIBK27Z6KHXHW6SRGYFSQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.849890456 (552h17m44.720973491s) [noise=true , address=145.79.227.3:4150 , piece_id=QVDXLVQH7Z2QQDHGRPSXAKJOPQDE2GSY3ONADAFEJZCDPEP7OETQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.853396733 (552h17m44.724479768s) [noise=true , address=145.79.227.2:4114 , piece_id=J7FC5V2NNJC267AYTTP2BR3DJE3U7UGFOEPBO2VNNO3AFSIZ3C7A, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.989310715 (552h17m44.86039376s) [noise=true , address=77.90.31.12:4590 , piece_id=LZOAI6GWSLU4HVHMUNAUWD7XD4OWRW3TMM6LV3BFMCIQRBOUCLXQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:45.109910548 (552h17m44.980993583s) [noise=true , address=77.90.31.10:4520 , piece_id=T5DZHFA3K4QZI55NTSUWXEQEE4EFAP2AM2CHWO5BXZC5V47QUOLQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:45.182016379 (552h17m45.053099414s) [noise=true , address=77.90.31.13:4629 , piece_id=VXT5RIWNO4YY5PPXECL24J3Y2JIJGYBPMFE3TLOXHPRF5PB57JBQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:45.725747105 (552h17m45.59683014s) [noise=true , address=77.90.31.21:4880 , piece_id=EMFCHYE3JV4ZN6HGPTS44S2XE7UCJ4XX6VECRPU7NQLYL6MQOCHA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:45.725765800 (552h17m45.596848835s) [seg_pos={0 0}, upload=484225]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:46:45.726011879 (552h17m45.597094914s) [noise=true , address=145.79.227.40:4049 , piece_id=66GVXKFY5J6YE6KAFID7ETGHJIDPEK7K775AGBSONIF6XGILIMFA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 28 15:46:45.726021828 (552h17m45.597104863s) [noise=true , address=145.79.227.40:4049 , piece_id=66GVXKFY5J6YE6KAFID7ETGHJIDPEK7K775AGBSONIF6XGILIMFA, seg_pos={0 0}, upload=484225]: Long tail cancellation due to optimal threshold
Dec 28 15:46:45.726109043 (552h17m45.597192079s) [noise=true , address=77.90.31.19:4844 , piece_id=BQPHVJES2H6GAVKFMU6EPJSJHCXIVAFOJ5AFA6FUTFOR2DUMNPOA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 28 15:46:45.726125305 (552h17m45.59720834s) [noise=true , address=77.90.31.19:4844 , piece_id=BQPHVJES2H6GAVKFMU6EPJSJHCXIVAFOJ5AFA6FUTFOR2DUMNPOA, seg_pos={0 0}, upload=484225]: Long tail cancellation due to optimal threshold
Dec 28 15:46:45.726145483 (552h17m45.597228518s) [noise=true , address=145.79.227.49:4339 , piece_id=HTQE56GE6N6KBML2FVFXM3JE7Q6FO7FKESJIC6JTE5KHBQB7KU7Q, seg_pos={0 0}, upload=484225]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 28 15:46:45.726151856 (552h17m45.597234891s) [noise=true , address=145.79.227.49:4339 , piece_id=HTQE56GE6N6KBML2FVFXM3JE7Q6FO7FKESJIC6JTE5KHBQB7KU7Q, seg_pos={0 0}, upload=484225]: Long tail cancellation due to optimal threshold
Dec 28 15:46:45.726170340 (552h17m45.597253375s) [seg_pos={0 0}, upload=484225]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:46:45.726236677 (552h17m45.597319712s) [upload=484225]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:46:45.726241606 (552h17m45.597324641s) [upload=484225]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:46:46.016393204 (552h17m45.887476249s) [upload=484225]: Done uploading segments...
Dec 28 15:46:46.016418120 (552h17m45.887501166s) [upload=484225]: Upload finished. err: <nil>
Duration: 5.906088364s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
16.707949681s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 15:47:44 UTC)
orbiter warning: context deadline exceeded
Duration: 16.707976622s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
39 out of 60 is failed (threshold=6) [XX.X.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX..XX...XXX....] duration:
3.517439415s trace:
https://jaeger.dev.storj.io/trace/64d166e63eaa7bab
FAIL (2025-12-28 15:47:36 UTC)
orbiter warning: context deadline exceeded
Duration: 3.517466356s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..................X....X...X........................X.......] duration:
28.326792634s trace:
https://jaeger.dev.storj.io/trace/3cc795a000779b86
FAIL (2025-12-28 15:47:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ) context deadline exceeded; piecestore: (Node ID: 128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, Piece ID: GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q) context deadline exceeded; piecestore: (Node ID: 12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, Piece ID: TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A) context deadline exceeded
======LOGS======
48.156469147 (552h11m48.027552193s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157156416 (552h11m48.028239461s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157556748 (552h11m48.028639793s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157945468 (552h11m48.029028503s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158645151 (552h11m48.029728196s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158842136 (552h11m48.029925171s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158983275 (552h11m48.03006631s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159344152 (552h11m48.030427187s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159350725 (552h11m48.03043376s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159710760 (552h11m48.030793795s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159732391 (552h11m48.030815437s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160068310 (552h11m48.031151346s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160439438 (552h11m48.031522473s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160813179 (552h11m48.031896225s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160839249 (552h11m48.031922284s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161271813 (552h11m48.032354848s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161765813 (552h11m48.032848848s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:49.456221594 (552h11m49.327304629s) [upload=484135]: Next returned nil segment
Dec 28 15:40:49.456235460 (552h11m49.327318495s) [upload=484135]: Waiting for error group managing segments...
Dec 28 15:40:49.789381987 (552h11m49.660465032s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.818159755 (552h11m49.6892428s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.901730607 (552h11m49.772813642s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.297735267 (552h11m50.168818312s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.866598958 (552h11m50.737682003s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.955467169 (552h11m50.826550214s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.029562809 (552h11m50.900645854s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.033202229 (552h11m50.904285264s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.119305864 (552h11m50.990388909s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.271054029 (552h11m51.142137074s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.374827580 (552h11m51.245910626s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.380607978 (552h11m51.251691013s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.500440018 (552h11m51.371523063s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688250240 (552h11m51.559333285s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688275438 (552h11m51.559358473s) [seg_pos={0 0}, upload=484135]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:40:51.688501768 (552h11m51.559584804s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688512109 (552h11m51.559595144s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688551694 (552h11m51.559634729s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688560741 (552h11m51.559643786s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688768767 (552h11m51.559851802s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWqAGD7Vrj2yzEVyNbfPp278Gga42xPLyAcdzGvUSuz3EgJYh): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688776031 (552h11m51.559859066s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688811729 (552h11m51.559894784s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688820707 (552h11m51.559903752s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688840654 (552h11m51.559923689s) [seg_pos={0 0}, upload=484135]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:40:51.688958800 (552h11m51.560041845s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:40:51.688965582 (552h11m51.560048617s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:40:51.990040599 (552h11m51.861123664s) [upload=484135]: Done uploading segments...
Dec 28 15:40:51.990091757 (552h11m51.861174802s) [upload=484135]: Upload finished. err: <nil>
Duration: 28.326811912s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [.X.X.X.....X......XX.X..XXX.X.XXX.XX.XX...X.X........X.X.X.X] duration:
30.002115301s trace:
https://jaeger.dev.storj.io/trace/6fd306ab031b44ee
FAIL (2025-12-28 15:47:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: LZOAI6GWSLU4HVHMUNAUWD7XD4OWRW3TMM6LV3BFMCIQRBOUCLXQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: T5DZHFA3K4QZI55NTSUWXEQEE4EFAP2AM2CHWO5BXZC5V47QUOLQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: VXT5RIWNO4YY5PPXECL24J3Y2JIJGYBPMFE3TLOXHPRF5PB57JBQ) context deadline exceeded
======LOGS======
Dec 28 15:46:42.551758296 (552h17m42.422841342s) [upload=484225]: Starting upload
Dec 28 15:46:42.551915806 (552h17m42.422998851s) [upload=484225]: Done starting upload
Dec 28 15:46:42.552009895 (552h17m42.42309293s) [upload=484225]: Uploading segments...
Dec 28 15:46:42.552240514 (552h17m42.423323559s) [upload=484225]: Got next segment. Inline: false
Dec 28 15:46:42.552306449 (552h17m42.423389495s) [upload=484225]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:46:42.552315588 (552h17m42.423398633s) [upload=484225]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:46:42.816464651 (552h17m42.687547696s) [seg_pos={0 0}, upload=484225]: Begin upload segment...
Dec 28 15:46:42.816519366 (552h17m42.687602401s) [seg_pos={0 0}, upload=484225]: Done begin upload segment.
Dec 28 15:46:42.816551005 (552h17m42.68763404s) [noise=true , address=77.90.31.12:4590 , piece_id=LZOAI6GWSLU4HVHMUNAUWD7XD4OWRW3TMM6LV3BFMCIQRBOUCLXQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.816645595 (552h17m42.68772863s) [noise=true , address=145.79.227.42:4122 , piece_id=VMFRYTCHGHPCWX7VVASLZNOYO42MF6GKIBK27Z6KHXHW6SRGYFSQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.817258433 (552h17m42.688341478s) [noise=true , address=145.79.227.19:4078 , piece_id=UXZJGEGBHGENUT65WA3ASHMDAYFTUNXSDDL6SLGOZCTNK7GXNR6A, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.817645119 (552h17m42.688728154s) [noise=true , address=77.90.31.10:4520 , piece_id=T5DZHFA3K4QZI55NTSUWXEQEE4EFAP2AM2CHWO5BXZC5V47QUOLQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.817889093 (552h17m42.688972128s) [noise=true , address=77.90.31.13:4629 , piece_id=VXT5RIWNO4YY5PPXECL24J3Y2JIJGYBPMFE3TLOXHPRF5PB57JBQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818067152 (552h17m42.689150187s) [noise=true , address=145.79.227.6:4259 , piece_id=VLDEVIATMLTPW7C4YMDKUR5DLXOXXMERLCWE4RN6HDERHXJ53VVA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818338649 (552h17m42.689421685s) [noise=true , address=145.79.227.7:4295 , piece_id=SY2PT47VSJ2TID4Y6XC7TCX3TCPTV5BTYV423ZJ2LZH4JVJC5ROQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818476963 (552h17m42.689559998s) [noise=true , address=77.90.31.21:4880 , piece_id=EMFCHYE3JV4ZN6HGPTS44S2XE7UCJ4XX6VECRPU7NQLYL6MQOCHA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818660542 (552h17m42.689743577s) [noise=true , address=145.79.227.27:4691 , piece_id=WYEVTHSUNDPR2EAO527L3XXGR7CJ6JKY7DMCYEDJOT3AUXKZ7WFQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.818916569 (552h17m42.689999604s) [noise=true , address=145.79.227.40:4049 , piece_id=66GVXKFY5J6YE6KAFID7ETGHJIDPEK7K775AGBSONIF6XGILIMFA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.819021549 (552h17m42.690104584s) [noise=true , address=145.79.227.3:4150 , piece_id=QVDXLVQH7Z2QQDHGRPSXAKJOPQDE2GSY3ONADAFEJZCDPEP7OETQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.819163069 (552h17m42.690246114s) [noise=true , address=77.90.31.19:4844 , piece_id=BQPHVJES2H6GAVKFMU6EPJSJHCXIVAFOJ5AFA6FUTFOR2DUMNPOA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.819627434 (552h17m42.690710469s) [noise=true , address=145.79.227.15:4875 , piece_id=N4XV5SACDNTTF7KXMRQUOF77LL7HFXWCJIL5YZFLBPYT5CPQF2RQ, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.820097719 (552h17m42.691180764s) [noise=true , address=145.79.227.41:4086 , piece_id=5XSLZQVEEFWB6ZEDVZZQMI2Q2KESZCEOFIRK6YEIGAJCJY3H7WGA, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.820517368 (552h17m42.691600413s) [noise=true , address=145.79.227.2:4114 , piece_id=J7FC5V2NNJC267AYTTP2BR3DJE3U7UGFOEPBO2VNNO3AFSIZ3C7A, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:42.821041806 (552h17m42.692124851s) [noise=true , address=145.79.227.49:4339 , piece_id=HTQE56GE6N6KBML2FVFXM3JE7Q6FO7FKESJIC6JTE5KHBQB7KU7Q, seg_pos={0 0}, upload=484225]: Uploading piece...
Dec 28 15:46:44.499663946 (552h17m44.370746991s) [upload=484225]: Next returned nil segment
Dec 28 15:46:44.499678063 (552h17m44.370761098s) [upload=484225]: Waiting for error group managing segments...
Dec 28 15:46:44.821148457 (552h17m44.692231502s) [noise=true , address=145.79.227.6:4259 , piece_id=VLDEVIATMLTPW7C4YMDKUR5DLXOXXMERLCWE4RN6HDERHXJ53VVA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.825878193 (552h17m44.696961228s) [noise=true , address=145.79.227.27:4691 , piece_id=WYEVTHSUNDPR2EAO527L3XXGR7CJ6JKY7DMCYEDJOT3AUXKZ7WFQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.826120755 (552h17m44.69720379s) [noise=true , address=145.79.227.19:4078 , piece_id=UXZJGEGBHGENUT65WA3ASHMDAYFTUNXSDDL6SLGOZCTNK7GXNR6A, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.829707705 (552h17m44.70079075s) [noise=true , address=145.79.227.41:4086 , piece_id=5XSLZQVEEFWB6ZEDVZZQMI2Q2KESZCEOFIRK6YEIGAJCJY3H7WGA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.836299617 (552h17m44.707382652s) [noise=true , address=145.79.227.15:4875 , piece_id=N4XV5SACDNTTF7KXMRQUOF77LL7HFXWCJIL5YZFLBPYT5CPQF2RQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.836585912 (552h17m44.707668957s) [noise=true , address=145.79.227.7:4295 , piece_id=SY2PT47VSJ2TID4Y6XC7TCX3TCPTV5BTYV423ZJ2LZH4JVJC5ROQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.846605191 (552h17m44.717688226s) [noise=true , address=145.79.227.42:4122 , piece_id=VMFRYTCHGHPCWX7VVASLZNOYO42MF6GKIBK27Z6KHXHW6SRGYFSQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.849890456 (552h17m44.720973491s) [noise=true , address=145.79.227.3:4150 , piece_id=QVDXLVQH7Z2QQDHGRPSXAKJOPQDE2GSY3ONADAFEJZCDPEP7OETQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.853396733 (552h17m44.724479768s) [noise=true , address=145.79.227.2:4114 , piece_id=J7FC5V2NNJC267AYTTP2BR3DJE3U7UGFOEPBO2VNNO3AFSIZ3C7A, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:44.989310715 (552h17m44.86039376s) [noise=true , address=77.90.31.12:4590 , piece_id=LZOAI6GWSLU4HVHMUNAUWD7XD4OWRW3TMM6LV3BFMCIQRBOUCLXQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:45.109910548 (552h17m44.980993583s) [noise=true , address=77.90.31.10:4520 , piece_id=T5DZHFA3K4QZI55NTSUWXEQEE4EFAP2AM2CHWO5BXZC5V47QUOLQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:45.182016379 (552h17m45.053099414s) [noise=true , address=77.90.31.13:4629 , piece_id=VXT5RIWNO4YY5PPXECL24J3Y2JIJGYBPMFE3TLOXHPRF5PB57JBQ, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:45.725747105 (552h17m45.59683014s) [noise=true , address=77.90.31.21:4880 , piece_id=EMFCHYE3JV4ZN6HGPTS44S2XE7UCJ4XX6VECRPU7NQLYL6MQOCHA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: <nil>
Dec 28 15:46:45.725765800 (552h17m45.596848835s) [seg_pos={0 0}, upload=484225]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:46:45.726011879 (552h17m45.597094914s) [noise=true , address=145.79.227.40:4049 , piece_id=66GVXKFY5J6YE6KAFID7ETGHJIDPEK7K775AGBSONIF6XGILIMFA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 28 15:46:45.726021828 (552h17m45.597104863s) [noise=true , address=145.79.227.40:4049 , piece_id=66GVXKFY5J6YE6KAFID7ETGHJIDPEK7K775AGBSONIF6XGILIMFA, seg_pos={0 0}, upload=484225]: Long tail cancellation due to optimal threshold
Dec 28 15:46:45.726109043 (552h17m45.597192079s) [noise=true , address=77.90.31.19:4844 , piece_id=BQPHVJES2H6GAVKFMU6EPJSJHCXIVAFOJ5AFA6FUTFOR2DUMNPOA, seg_pos={0 0}, upload=484225]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 28 15:46:45.726125305 (552h17m45.59720834s) [noise=true , address=77.90.31.19:4844 , piece_id=BQPHVJES2H6GAVKFMU6EPJSJHCXIVAFOJ5AFA6FUTFOR2DUMNPOA, seg_pos={0 0}, upload=484225]: Long tail cancellation due to optimal threshold
Dec 28 15:46:45.726145483 (552h17m45.597228518s) [noise=true , address=145.79.227.49:4339 , piece_id=HTQE56GE6N6KBML2FVFXM3JE7Q6FO7FKESJIC6JTE5KHBQB7KU7Q, seg_pos={0 0}, upload=484225]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 28 15:46:45.726151856 (552h17m45.597234891s) [noise=true , address=145.79.227.49:4339 , piece_id=HTQE56GE6N6KBML2FVFXM3JE7Q6FO7FKESJIC6JTE5KHBQB7KU7Q, seg_pos={0 0}, upload=484225]: Long tail cancellation due to optimal threshold
Dec 28 15:46:45.726170340 (552h17m45.597253375s) [seg_pos={0 0}, upload=484225]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:46:45.726236677 (552h17m45.597319712s) [upload=484225]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:46:45.726241606 (552h17m45.597324641s) [upload=484225]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:46:46.016393204 (552h17m45.887476249s) [upload=484225]: Done uploading segments...
Dec 28 15:46:46.016418120 (552h17m45.887501166s) [upload=484225]: Upload finished. err: <nil>
Duration: 30.002139487s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [XXX.X.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX..XX...XXX...] duration:
3.534216842s trace:
https://jaeger.dev.storj.io/trace/64d166e63eaa7bab
FAIL (2025-12-28 15:47:10 UTC)
orbiter warning: context deadline exceeded
Duration: 3.534245576s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............X................................X..............] duration:
15.404989386s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:47:02 UTC)
orbiter warning: context deadline exceeded
Duration: 15.405020244s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [.XXX.X.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX..XX...XXX..] duration:
3.57179685s trace:
https://jaeger.dev.storj.io/trace/64d166e63eaa7bab
FAIL (2025-12-28 15:46:51 UTC)
orbiter warning: context deadline exceeded
Duration: 3.571828379s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...................X....X...X........................X......] duration:
49.088438154s trace:
https://jaeger.dev.storj.io/trace/3cc795a000779b86
FAIL (2025-12-28 15:46:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ) context deadline exceeded; piecestore: (Node ID: 128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, Piece ID: GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q) context deadline exceeded; piecestore: (Node ID: 12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, Piece ID: TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A) context deadline exceeded
======LOGS======
48.156469147 (552h11m48.027552193s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157156416 (552h11m48.028239461s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157556748 (552h11m48.028639793s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157945468 (552h11m48.029028503s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158645151 (552h11m48.029728196s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158842136 (552h11m48.029925171s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158983275 (552h11m48.03006631s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159344152 (552h11m48.030427187s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159350725 (552h11m48.03043376s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159710760 (552h11m48.030793795s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159732391 (552h11m48.030815437s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160068310 (552h11m48.031151346s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160439438 (552h11m48.031522473s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160813179 (552h11m48.031896225s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160839249 (552h11m48.031922284s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161271813 (552h11m48.032354848s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161765813 (552h11m48.032848848s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:49.456221594 (552h11m49.327304629s) [upload=484135]: Next returned nil segment
Dec 28 15:40:49.456235460 (552h11m49.327318495s) [upload=484135]: Waiting for error group managing segments...
Dec 28 15:40:49.789381987 (552h11m49.660465032s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.818159755 (552h11m49.6892428s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.901730607 (552h11m49.772813642s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.297735267 (552h11m50.168818312s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.866598958 (552h11m50.737682003s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.955467169 (552h11m50.826550214s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.029562809 (552h11m50.900645854s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.033202229 (552h11m50.904285264s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.119305864 (552h11m50.990388909s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.271054029 (552h11m51.142137074s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.374827580 (552h11m51.245910626s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.380607978 (552h11m51.251691013s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.500440018 (552h11m51.371523063s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688250240 (552h11m51.559333285s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688275438 (552h11m51.559358473s) [seg_pos={0 0}, upload=484135]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:40:51.688501768 (552h11m51.559584804s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688512109 (552h11m51.559595144s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688551694 (552h11m51.559634729s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688560741 (552h11m51.559643786s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688768767 (552h11m51.559851802s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWqAGD7Vrj2yzEVyNbfPp278Gga42xPLyAcdzGvUSuz3EgJYh): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688776031 (552h11m51.559859066s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688811729 (552h11m51.559894784s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688820707 (552h11m51.559903752s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688840654 (552h11m51.559923689s) [seg_pos={0 0}, upload=484135]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:40:51.688958800 (552h11m51.560041845s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:40:51.688965582 (552h11m51.560048617s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:40:51.990040599 (552h11m51.861123664s) [upload=484135]: Done uploading segments...
Dec 28 15:40:51.990091757 (552h11m51.861174802s) [upload=484135]: Upload finished. err: <nil>
Duration: 49.08846741s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............X................................X.............] duration:
9.771755081s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:46:32 UTC)
orbiter warning: context deadline exceeded
Duration: 9.771782092s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
10.93644199s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 15:46:28 UTC)
orbiter warning: context deadline exceeded
Duration: 10.936461687s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [X.XXX.X.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX..XX...XXX.] duration:
3.577493187s trace:
https://jaeger.dev.storj.io/trace/64d166e63eaa7bab
FAIL (2025-12-28 15:45:56 UTC)
orbiter warning: context deadline exceeded
Duration: 3.577569393s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [....................X....X...X........................X.....] duration:
39.399067859s trace:
https://jaeger.dev.storj.io/trace/3cc795a000779b86
FAIL (2025-12-28 15:45:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ) context deadline exceeded; piecestore: (Node ID: 128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, Piece ID: GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q) context deadline exceeded; piecestore: (Node ID: 12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, Piece ID: TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A) context deadline exceeded
======LOGS======
48.156469147 (552h11m48.027552193s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157156416 (552h11m48.028239461s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157556748 (552h11m48.028639793s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157945468 (552h11m48.029028503s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158645151 (552h11m48.029728196s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158842136 (552h11m48.029925171s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158983275 (552h11m48.03006631s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159344152 (552h11m48.030427187s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159350725 (552h11m48.03043376s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159710760 (552h11m48.030793795s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159732391 (552h11m48.030815437s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160068310 (552h11m48.031151346s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160439438 (552h11m48.031522473s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160813179 (552h11m48.031896225s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160839249 (552h11m48.031922284s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161271813 (552h11m48.032354848s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161765813 (552h11m48.032848848s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:49.456221594 (552h11m49.327304629s) [upload=484135]: Next returned nil segment
Dec 28 15:40:49.456235460 (552h11m49.327318495s) [upload=484135]: Waiting for error group managing segments...
Dec 28 15:40:49.789381987 (552h11m49.660465032s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.818159755 (552h11m49.6892428s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.901730607 (552h11m49.772813642s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.297735267 (552h11m50.168818312s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.866598958 (552h11m50.737682003s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.955467169 (552h11m50.826550214s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.029562809 (552h11m50.900645854s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.033202229 (552h11m50.904285264s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.119305864 (552h11m50.990388909s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.271054029 (552h11m51.142137074s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.374827580 (552h11m51.245910626s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.380607978 (552h11m51.251691013s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.500440018 (552h11m51.371523063s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688250240 (552h11m51.559333285s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688275438 (552h11m51.559358473s) [seg_pos={0 0}, upload=484135]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:40:51.688501768 (552h11m51.559584804s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688512109 (552h11m51.559595144s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688551694 (552h11m51.559634729s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688560741 (552h11m51.559643786s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688768767 (552h11m51.559851802s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWqAGD7Vrj2yzEVyNbfPp278Gga42xPLyAcdzGvUSuz3EgJYh): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688776031 (552h11m51.559859066s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688811729 (552h11m51.559894784s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688820707 (552h11m51.559903752s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688840654 (552h11m51.559923689s) [seg_pos={0 0}, upload=484135]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:40:51.688958800 (552h11m51.560041845s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:40:51.688965582 (552h11m51.560048617s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:40:51.990040599 (552h11m51.861123664s) [upload=484135]: Done uploading segments...
Dec 28 15:40:51.990091757 (552h11m51.861174802s) [upload=484135]: Upload finished. err: <nil>
Duration: 39.399088699s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [..X.X.X.....X......XX.X..XXX.X.XXX.XX.XX...X.X........X.X.X.] duration:
6.052888016s trace:
https://jaeger.dev.storj.io/trace/354f5c93194630fe
FAIL (2025-12-28 15:45:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 6JZJDZI7FXSZW3H2R25DJYLS5GIZM2XJJIO6FCNN6Q6Y63V7TZOQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: KMQ7XW76YAJA5HDEKKFK22GHKMAZHECR6DKBRVDEBUJE63F6MMOQ) context deadline exceeded
======LOGS======
segments...
Dec 28 15:44:23.228864797 (552h15m23.099947852s) [upload=484187]: Got next segment. Inline: false
Dec 28 15:44:23.228965468 (552h15m23.100048503s) [upload=484187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:44:23.228972282 (552h15m23.100055317s) [upload=484187]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:44:23.473522066 (552h15m23.344605101s) [seg_pos={0 0}, upload=484187]: Begin upload segment...
Dec 28 15:44:23.473581239 (552h15m23.344664274s) [seg_pos={0 0}, upload=484187]: Done begin upload segment.
Dec 28 15:44:23.473646121 (552h15m23.344729156s) [noise=true , address=145.79.227.32:4834 , piece_id=XWAALBLYIILFWOWOMHB3FM6LNKQLNSDTQQ7IGWJLI73J7S4T2BPA, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.473663134 (552h15m23.344746169s) [noise=true , address=45.91.116.6:4974 , piece_id=GQN5ZGZBHTVMGJIF25WZVAUGRYVDVESYCPRAORIVMTKFELUD4NYQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.474181331 (552h15m23.345264376s) [noise=true , address=45.91.116.5:4300 , piece_id=6VVNTJLAH6KTCNUCJBUJMDVWZXFJQJQ3EXXCNSDSDCESSPQBVFOQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.474675652 (552h15m23.345758687s) [noise=true , address=145.79.227.38:4979 , piece_id=ICQUTWM7KWDEGV6MQVLRNMHL4B55EEBI6IILHUZ32DQIKLIIRZEA, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.475073770 (552h15m23.346156805s) [noise=true , address=145.79.227.18:4007 , piece_id=YVNQUVKS4NHHSQYUDNHUNVLNFQITS56QLX77DFDGXBGT2XNKI7KQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.475456669 (552h15m23.346539734s) [noise=true , address=145.79.227.45:4231 , piece_id=PTWRS7JWHN4NKEU7JBHTPU272X6L7KFX7Z5IVQJ6FZSHVQGAJZOQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.475465355 (552h15m23.3465484s) [noise=true , address=77.90.31.10:4519 , piece_id=K2R5W7KM64SQCMS7RHKFBSBKJDUOCTSVJ6PRIX5BRCVMPOI55M4Q, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.475866389 (552h15m23.346949434s) [noise=true , address=145.79.227.48:4302 , piece_id=Y7ZIYCSKVNRTQYDS2JAVEBPFIDUGUVUPSBUJ5VDAH7FOR6QOC54A, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.476195155 (552h15m23.34727819s) [noise=true , address=145.79.227.52:4445 , piece_id=DWK5YFNHIHKITVDLWI3VZVAAAHNX67RZHVIVYLU2FEC3IZB3QXGA, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.476489416 (552h15m23.347572461s) [noise=true , address=145.79.227.51:4409 , piece_id=V4BNGIDOFEZ2PS6HHMAZFQSMUXZT6DVLHE6U3CMFPBPGR2DMLZGQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.476768978 (552h15m23.347852013s) [noise=true , address=45.91.116.12:4228 , piece_id=6JZJDZI7FXSZW3H2R25DJYLS5GIZM2XJJIO6FCNN6Q6Y63V7TZOQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.477082856 (552h15m23.348165891s) [noise=true , address=145.79.227.47:4266 , piece_id=KMT6CODJY2TIU4LRVVH7EBNR5W43IJPTSQHQNSTCM665KY4BON4A, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.477730388 (552h15m23.348813433s) [noise=true , address=45.91.116.10:4083 , piece_id=KMQ7XW76YAJA5HDEKKFK22GHKMAZHECR6DKBRVDEBUJE63F6MMOQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.479783297 (552h15m23.350866342s) [noise=true , address=145.79.227.11:4731 , piece_id=YT3FOQNK2UL5M57HEZ6ESBTVMST2QS6WCRDLQ3MRVLMWPVWGWGWQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.482206953 (552h15m23.353289988s) [noise=true , address=45.91.116.7:4009 , piece_id=5YUBW6YCUMNKCGJGDZNIX4SE7D2MCIXPKNGZ7PDCP72U3X6PT33A, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.484736129 (552h15m23.355819164s) [noise=true , address=145.79.227.37:4567 , piece_id=TNL66LEKN2EAVIZXCJFML7BIEL2AR7OVD43Y3LDIEFBGVEPLVR7Q, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:25.195219194 (552h15m25.06630223s) [upload=484187]: Next returned nil segment
Dec 28 15:44:25.195233162 (552h15m25.066316197s) [upload=484187]: Waiting for error group managing segments...
Dec 28 15:44:25.479305062 (552h15m25.350388097s) [noise=true , address=145.79.227.18:4007 , piece_id=YVNQUVKS4NHHSQYUDNHUNVLNFQITS56QLX77DFDGXBGT2XNKI7KQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.479557773 (552h15m25.350640808s) [noise=true , address=145.79.227.38:4979 , piece_id=ICQUTWM7KWDEGV6MQVLRNMHL4B55EEBI6IILHUZ32DQIKLIIRZEA, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.495908758 (552h15m25.366991794s) [noise=true , address=145.79.227.11:4731 , piece_id=YT3FOQNK2UL5M57HEZ6ESBTVMST2QS6WCRDLQ3MRVLMWPVWGWGWQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.498335489 (552h15m25.369418524s) [noise=true , address=145.79.227.37:4567 , piece_id=TNL66LEKN2EAVIZXCJFML7BIEL2AR7OVD43Y3LDIEFBGVEPLVR7Q, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.498557702 (552h15m25.369640748s) [noise=true , address=145.79.227.47:4266 , piece_id=KMT6CODJY2TIU4LRVVH7EBNR5W43IJPTSQHQNSTCM665KY4BON4A, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.502042959 (552h15m25.373125994s) [noise=true , address=145.79.227.45:4231 , piece_id=PTWRS7JWHN4NKEU7JBHTPU272X6L7KFX7Z5IVQJ6FZSHVQGAJZOQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.507488137 (552h15m25.378571172s) [noise=true , address=145.79.227.51:4409 , piece_id=V4BNGIDOFEZ2PS6HHMAZFQSMUXZT6DVLHE6U3CMFPBPGR2DMLZGQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.512839918 (552h15m25.383922963s) [noise=true , address=145.79.227.52:4445 , piece_id=DWK5YFNHIHKITVDLWI3VZVAAAHNX67RZHVIVYLU2FEC3IZB3QXGA, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.514591643 (552h15m25.385674678s) [noise=true , address=145.79.227.48:4302 , piece_id=Y7ZIYCSKVNRTQYDS2JAVEBPFIDUGUVUPSBUJ5VDAH7FOR6QOC54A, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.528475671 (552h15m25.399558716s) [noise=true , address=145.79.227.32:4834 , piece_id=XWAALBLYIILFWOWOMHB3FM6LNKQLNSDTQQ7IGWJLI73J7S4T2BPA, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.949610562 (552h15m25.820693617s) [noise=true , address=45.91.116.12:4228 , piece_id=6JZJDZI7FXSZW3H2R25DJYLS5GIZM2XJJIO6FCNN6Q6Y63V7TZOQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:26.470746833 (552h15m26.341829878s) [noise=true , address=45.91.116.7:4009 , piece_id=5YUBW6YCUMNKCGJGDZNIX4SE7D2MCIXPKNGZ7PDCP72U3X6PT33A, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:26.557979107 (552h15m26.429062142s) [noise=true , address=45.91.116.10:4083 , piece_id=KMQ7XW76YAJA5HDEKKFK22GHKMAZHECR6DKBRVDEBUJE63F6MMOQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:26.557998314 (552h15m26.429081349s) [seg_pos={0 0}, upload=484187]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:44:26.558185981 (552h15m26.429269016s) [noise=true , address=45.91.116.5:4300 , piece_id=6VVNTJLAH6KTCNUCJBUJMDVWZXFJQJQ3EXXCNSDSDCESSPQBVFOQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 28 15:44:26.558193195 (552h15m26.429276241s) [noise=true , address=45.91.116.5:4300 , piece_id=6VVNTJLAH6KTCNUCJBUJMDVWZXFJQJQ3EXXCNSDSDCESSPQBVFOQ, seg_pos={0 0}, upload=484187]: Long tail cancellation due to optimal threshold
Dec 28 15:44:26.558306611 (552h15m26.429389656s) [noise=true , address=77.90.31.10:4519 , piece_id=K2R5W7KM64SQCMS7RHKFBSBKJDUOCTSVJ6PRIX5BRCVMPOI55M4Q, seg_pos={0 0}, upload=484187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:42294->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:42294->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42294->77.90.31.10:4519: use of closed network connection
Dec 28 15:44:26.558312673 (552h15m26.429395708s) [noise=true , address=77.90.31.10:4519 , piece_id=K2R5W7KM64SQCMS7RHKFBSBKJDUOCTSVJ6PRIX5BRCVMPOI55M4Q, seg_pos={0 0}, upload=484187]: Long tail cancellation due to optimal threshold
Dec 28 15:44:26.558556727 (552h15m26.429639782s) [noise=true , address=45.91.116.6:4974 , piece_id=GQN5ZGZBHTVMGJIF25WZVAUGRYVDVESYCPRAORIVMTKFELUD4NYQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52484->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52484->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52484->45.91.116.6:4974: use of closed network connection
Dec 28 15:44:26.558577006 (552h15m26.429660061s) [noise=true , address=45.91.116.6:4974 , piece_id=GQN5ZGZBHTVMGJIF25WZVAUGRYVDVESYCPRAORIVMTKFELUD4NYQ, seg_pos={0 0}, upload=484187]: Long tail cancellation due to optimal threshold
Dec 28 15:44:26.558613987 (552h15m26.429697022s) [seg_pos={0 0}, upload=484187]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:44:26.558755165 (552h15m26.4298382s) [upload=484187]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:44:26.558762529 (552h15m26.429845564s) [upload=484187]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:44:26.851863402 (552h15m26.722946437s) [upload=484187]: Done uploading segments...
Dec 28 15:44:26.851883069 (552h15m26.722966114s) [upload=484187]: Upload finished. err: <nil>
Duration: 6.052947028s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
14.822894156s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 15:45:28 UTC)
orbiter warning: context deadline exceeded
Duration: 14.8229222s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............X................................X............] duration:
15.229963572s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:45:23 UTC)
orbiter warning: context deadline exceeded
Duration: 15.229988419s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....................X....X...X........................X....] duration:
42.668209424s trace:
https://jaeger.dev.storj.io/trace/3cc795a000779b86
FAIL (2025-12-28 15:45:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ) context deadline exceeded; piecestore: (Node ID: 128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, Piece ID: GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q) context deadline exceeded; piecestore: (Node ID: 12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, Piece ID: TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A) context deadline exceeded
======LOGS======
48.156469147 (552h11m48.027552193s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157156416 (552h11m48.028239461s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157556748 (552h11m48.028639793s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157945468 (552h11m48.029028503s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158645151 (552h11m48.029728196s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158842136 (552h11m48.029925171s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158983275 (552h11m48.03006631s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159344152 (552h11m48.030427187s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159350725 (552h11m48.03043376s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159710760 (552h11m48.030793795s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159732391 (552h11m48.030815437s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160068310 (552h11m48.031151346s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160439438 (552h11m48.031522473s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160813179 (552h11m48.031896225s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160839249 (552h11m48.031922284s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161271813 (552h11m48.032354848s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161765813 (552h11m48.032848848s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:49.456221594 (552h11m49.327304629s) [upload=484135]: Next returned nil segment
Dec 28 15:40:49.456235460 (552h11m49.327318495s) [upload=484135]: Waiting for error group managing segments...
Dec 28 15:40:49.789381987 (552h11m49.660465032s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.818159755 (552h11m49.6892428s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.901730607 (552h11m49.772813642s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.297735267 (552h11m50.168818312s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.866598958 (552h11m50.737682003s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.955467169 (552h11m50.826550214s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.029562809 (552h11m50.900645854s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.033202229 (552h11m50.904285264s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.119305864 (552h11m50.990388909s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.271054029 (552h11m51.142137074s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.374827580 (552h11m51.245910626s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.380607978 (552h11m51.251691013s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.500440018 (552h11m51.371523063s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688250240 (552h11m51.559333285s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688275438 (552h11m51.559358473s) [seg_pos={0 0}, upload=484135]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:40:51.688501768 (552h11m51.559584804s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688512109 (552h11m51.559595144s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688551694 (552h11m51.559634729s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688560741 (552h11m51.559643786s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688768767 (552h11m51.559851802s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWqAGD7Vrj2yzEVyNbfPp278Gga42xPLyAcdzGvUSuz3EgJYh): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688776031 (552h11m51.559859066s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688811729 (552h11m51.559894784s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688820707 (552h11m51.559903752s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688840654 (552h11m51.559923689s) [seg_pos={0 0}, upload=484135]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:40:51.688958800 (552h11m51.560041845s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:40:51.688965582 (552h11m51.560048617s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:40:51.990040599 (552h11m51.861123664s) [upload=484135]: Done uploading segments...
Dec 28 15:40:51.990091757 (552h11m51.861174802s) [upload=484135]: Upload finished. err: <nil>
Duration: 42.668230714s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX..XX...XXX] duration:
5.00134606s trace:
https://jaeger.dev.storj.io/trace/64d166e63eaa7bab
FAIL (2025-12-28 15:44:59 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001369604s
core
uplink
us-select-1
us1-placement-20
|