Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 9.359008133s trace: https://jaeger.dev.storj.io/trace/65df260343fa2062 FAIL (2025-12-05 14:13:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded ======LOGS====== Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments... Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment... Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment. Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments... Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces. Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments... Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil> Duration: 9.35903796s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 13.202716163s trace: https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6 FAIL (2025-12-05 14:13:50 UTC) orbiter warning: context deadline exceeded Duration: 13.202745809s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 7.494725506s trace: https://jaeger.dev.storj.io/trace/70e54e58d52fa95e FAIL (2025-12-05 14:13:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, Piece ID: MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q) context deadline exceeded; piecestore: (Node ID: 12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, Piece ID: UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ) context deadline exceeded; piecestore: (Node ID: 12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4, Piece ID: 4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA) context deadline exceeded; piecestore: (Node ID: 12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, Piece ID: SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA) context deadline exceeded; piecestore: (Node ID: 123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS, Piece ID: MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA) context deadline exceeded; piecestore: (Node ID: 121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, Piece ID: EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ) context deadline exceeded; piecestore: (Node ID: 1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY, Piece ID: CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA) context deadline exceeded ======LOGS====== 4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669180895 (45h13m48.140391634s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669641381 (45h13m48.14085212s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670111447 (45h13m48.141322196s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670601259 (45h13m48.141811999s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671075502 (45h13m48.142286241s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671598037 (45h13m48.142808776s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672127346 (45h13m48.143338105s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672648588 (45h13m48.143859327s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673155192 (45h13m48.144365931s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673691684 (45h13m48.144902423s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.674805966 (45h13m48.146016705s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.675360772 (45h13m48.146571511s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:13.239845356 (45h13m49.711056085s) [upload=39772]: Next returned nil segment Dec 5 14:09:13.239860604 (45h13m49.711071334s) [upload=39772]: Waiting for error group managing segments... Dec 5 14:09:13.643777330 (45h13m50.114988069s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.697999800 (45h13m50.169210539s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.713127946 (45h13m50.184338685s) [noise=true , address=77.90.31.15:4729 , piece_id=IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.718498752 (45h13m50.189709491s) [noise=true , address=77.90.31.21:4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.726522451 (45h13m50.19773318s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.749057442 (45h13m50.220268171s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.750808776 (45h13m50.222019515s) [noise=true , address=77.90.31.10:4542 , piece_id=XDD4QZAKCWJK5A3F5FDL4NRNPEQZF3GKODCGVT2V47AH6KVEOOWQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.754762756 (45h13m50.225973485s) [noise=true , address=77.90.31.11:4556 , piece_id=WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.759242185 (45h13m50.230452925s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.774028982 (45h13m50.245239711s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.786684680 (45h13m50.257895419s) [noise=true , address=45.91.116.9:4272 , piece_id=4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.790669208 (45h13m50.261879947s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.850747877 (45h13m50.321958617s) [noise=true , address=45.91.116.8:4071 , piece_id=3YI67CEXFW75VRROYLSABR55DNVFHE4SR6EUP57CPRIIB63GZOVA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884817071 (45h13m50.3560278s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884838361 (45h13m50.35604909s) [seg_pos={0 0}, upload=39772]: Segment reached optimal threshold of 14 pieces. Dec 5 14:09:13.885054584 (45h13m50.356265313s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3): protocol: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection Dec 5 14:09:13.885064302 (45h13m50.356275041s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885416693 (45h13m50.356627432s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection Dec 5 14:09:13.885427744 (45h13m50.356638483s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885527824 (45h13m50.356738563s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection Dec 5 14:09:13.885535649 (45h13m50.356746388s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885614951 (45h13m50.35682569s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection Dec 5 14:09:13.885620010 (45h13m50.356830749s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885636241 (45h13m50.35684697s) [seg_pos={0 0}, upload=39772]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 5 14:09:13.885731973 (45h13m50.356942722s) [upload=39772]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 14:09:13.885739618 (45h13m50.356950357s) [upload=39772]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:09:14.639752295 (45h13m51.110963024s) [upload=39772]: Done uploading segments... Dec 5 14:09:14.639788193 (45h13m51.110998932s) [upload=39772]: Upload finished. err: <nil> Duration: 7.494816589s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 4.902166362s trace: https://jaeger.dev.storj.io/trace/41b9575307efc1e3 FAIL (2025-12-05 14:13:33 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.902189236s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X..X.......X................................] duration: 8.56534774s trace: https://jaeger.dev.storj.io/trace/5cac00e234ac2601 FAIL (2025-12-05 14:13:20 UTC) orbiter warning: context deadline exceeded Duration: 8.565368159s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [XXX.XXXX.XXX.XXXXXXXXXXXXXXXXXXXXXX.X..XXXXX..XXXXXX.X.XXXXX] duration: 5.000820518s trace: https://jaeger.dev.storj.io/trace/32e127d22c78410b FAIL (2025-12-05 14:13:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000834564s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 9.447632463s trace: https://jaeger.dev.storj.io/trace/70e54e58d52fa95e FAIL (2025-12-05 14:13:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, Piece ID: MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q) context deadline exceeded; piecestore: (Node ID: 12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, Piece ID: UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ) context deadline exceeded; piecestore: (Node ID: 12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4, Piece ID: 4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA) context deadline exceeded; piecestore: (Node ID: 12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, Piece ID: SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA) context deadline exceeded; piecestore: (Node ID: 123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS, Piece ID: MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA) context deadline exceeded; piecestore: (Node ID: 121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, Piece ID: EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ) context deadline exceeded; piecestore: (Node ID: 1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY, Piece ID: CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA) context deadline exceeded ======LOGS====== 4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669180895 (45h13m48.140391634s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669641381 (45h13m48.14085212s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670111447 (45h13m48.141322196s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670601259 (45h13m48.141811999s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671075502 (45h13m48.142286241s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671598037 (45h13m48.142808776s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672127346 (45h13m48.143338105s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672648588 (45h13m48.143859327s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673155192 (45h13m48.144365931s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673691684 (45h13m48.144902423s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.674805966 (45h13m48.146016705s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.675360772 (45h13m48.146571511s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:13.239845356 (45h13m49.711056085s) [upload=39772]: Next returned nil segment Dec 5 14:09:13.239860604 (45h13m49.711071334s) [upload=39772]: Waiting for error group managing segments... Dec 5 14:09:13.643777330 (45h13m50.114988069s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.697999800 (45h13m50.169210539s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.713127946 (45h13m50.184338685s) [noise=true , address=77.90.31.15:4729 , piece_id=IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.718498752 (45h13m50.189709491s) [noise=true , address=77.90.31.21:4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.726522451 (45h13m50.19773318s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.749057442 (45h13m50.220268171s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.750808776 (45h13m50.222019515s) [noise=true , address=77.90.31.10:4542 , piece_id=XDD4QZAKCWJK5A3F5FDL4NRNPEQZF3GKODCGVT2V47AH6KVEOOWQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.754762756 (45h13m50.225973485s) [noise=true , address=77.90.31.11:4556 , piece_id=WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.759242185 (45h13m50.230452925s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.774028982 (45h13m50.245239711s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.786684680 (45h13m50.257895419s) [noise=true , address=45.91.116.9:4272 , piece_id=4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.790669208 (45h13m50.261879947s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.850747877 (45h13m50.321958617s) [noise=true , address=45.91.116.8:4071 , piece_id=3YI67CEXFW75VRROYLSABR55DNVFHE4SR6EUP57CPRIIB63GZOVA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884817071 (45h13m50.3560278s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884838361 (45h13m50.35604909s) [seg_pos={0 0}, upload=39772]: Segment reached optimal threshold of 14 pieces. Dec 5 14:09:13.885054584 (45h13m50.356265313s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3): protocol: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection Dec 5 14:09:13.885064302 (45h13m50.356275041s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885416693 (45h13m50.356627432s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection Dec 5 14:09:13.885427744 (45h13m50.356638483s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885527824 (45h13m50.356738563s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection Dec 5 14:09:13.885535649 (45h13m50.356746388s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885614951 (45h13m50.35682569s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection Dec 5 14:09:13.885620010 (45h13m50.356830749s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885636241 (45h13m50.35684697s) [seg_pos={0 0}, upload=39772]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 5 14:09:13.885731973 (45h13m50.356942722s) [upload=39772]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 14:09:13.885739618 (45h13m50.356950357s) [upload=39772]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:09:14.639752295 (45h13m51.110963024s) [upload=39772]: Done uploading segments... Dec 5 14:09:14.639788193 (45h13m51.110998932s) [upload=39772]: Upload finished. err: <nil> Duration: 9.447661788s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 3.390379175s trace: https://jaeger.dev.storj.io/trace/361a2ea17a62744e FAIL (2025-12-05 14:13:13 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 5 14:08:02.649750072 (45h12m39.120960811s) [upload=39762]: Starting upload Dec 5 14:08:02.650012922 (45h12m39.121223681s) [upload=39762]: Done starting upload Dec 5 14:08:02.650140385 (45h12m39.121351124s) [upload=39762]: Uploading segments... Dec 5 14:08:02.650204036 (45h12m39.121414775s) [upload=39762]: Got next segment. Inline: true Dec 5 14:08:02.650210659 (45h12m39.121421398s) [upload=39762]: Waiting for error group managing segments... Dec 5 14:08:02.650285611 (45h12m39.12149635s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 14:08:02.650292053 (45h12m39.121502792s) [upload=39762]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 5 14:08:02.650295811 (45h12m39.12150655s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:08:05.223576463 (45h12m41.694787212s) [upload=39762]: Done uploading segments... Dec 5 14:08:05.223604205 (45h12m41.694814934s) [upload=39762]: Upload finished. err: <nil> Duration:* 3.390418821s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 7.576662108s trace: https://jaeger.dev.storj.io/trace/572c91a59c970da7 FAIL (2025-12-05 14:13:13 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 7.576718394s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 13.782762274s trace: https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6 FAIL (2025-12-05 14:13:11 UTC) orbiter warning: context deadline exceeded Duration: 13.782804293s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 19.300092051s trace: https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6 FAIL (2025-12-05 14:12:57 UTC) orbiter warning: context deadline exceeded Duration: 19.300125234s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 9.762532844s trace: https://jaeger.dev.storj.io/trace/65df260343fa2062 FAIL (2025-12-05 14:12:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded ======LOGS====== Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments... Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment... Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment. Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments... Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces. Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments... Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil> Duration: 9.762557421s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.874892111s trace: https://jaeger.dev.storj.io/trace/361a2ea17a62744e FAIL (2025-12-05 14:12:48 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 5 14:08:02.649750072 (45h12m39.120960811s) [upload=39762]: Starting upload Dec 5 14:08:02.650012922 (45h12m39.121223681s) [upload=39762]: Done starting upload Dec 5 14:08:02.650140385 (45h12m39.121351124s) [upload=39762]: Uploading segments... Dec 5 14:08:02.650204036 (45h12m39.121414775s) [upload=39762]: Got next segment. Inline: true Dec 5 14:08:02.650210659 (45h12m39.121421398s) [upload=39762]: Waiting for error group managing segments... Dec 5 14:08:02.650285611 (45h12m39.12149635s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 14:08:02.650292053 (45h12m39.121502792s) [upload=39762]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 5 14:08:02.650295811 (45h12m39.12150655s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:08:05.223576463 (45h12m41.694787212s) [upload=39762]: Done uploading segments... Dec 5 14:08:05.223604205 (45h12m41.694814934s) [upload=39762]: Upload finished. err: <nil> Duration:* 2.874922028s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 9.209433991s trace: https://jaeger.dev.storj.io/trace/70e54e58d52fa95e FAIL (2025-12-05 14:12:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, Piece ID: MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q) context deadline exceeded; piecestore: (Node ID: 12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, Piece ID: UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ) context deadline exceeded; piecestore: (Node ID: 12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4, Piece ID: 4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA) context deadline exceeded; piecestore: (Node ID: 12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, Piece ID: SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA) context deadline exceeded; piecestore: (Node ID: 123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS, Piece ID: MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA) context deadline exceeded; piecestore: (Node ID: 121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, Piece ID: EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ) context deadline exceeded; piecestore: (Node ID: 1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY, Piece ID: CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA) context deadline exceeded ======LOGS====== 4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669180895 (45h13m48.140391634s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669641381 (45h13m48.14085212s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670111447 (45h13m48.141322196s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670601259 (45h13m48.141811999s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671075502 (45h13m48.142286241s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671598037 (45h13m48.142808776s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672127346 (45h13m48.143338105s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672648588 (45h13m48.143859327s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673155192 (45h13m48.144365931s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673691684 (45h13m48.144902423s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.674805966 (45h13m48.146016705s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.675360772 (45h13m48.146571511s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:13.239845356 (45h13m49.711056085s) [upload=39772]: Next returned nil segment Dec 5 14:09:13.239860604 (45h13m49.711071334s) [upload=39772]: Waiting for error group managing segments... Dec 5 14:09:13.643777330 (45h13m50.114988069s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.697999800 (45h13m50.169210539s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.713127946 (45h13m50.184338685s) [noise=true , address=77.90.31.15:4729 , piece_id=IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.718498752 (45h13m50.189709491s) [noise=true , address=77.90.31.21:4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.726522451 (45h13m50.19773318s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.749057442 (45h13m50.220268171s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.750808776 (45h13m50.222019515s) [noise=true , address=77.90.31.10:4542 , piece_id=XDD4QZAKCWJK5A3F5FDL4NRNPEQZF3GKODCGVT2V47AH6KVEOOWQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.754762756 (45h13m50.225973485s) [noise=true , address=77.90.31.11:4556 , piece_id=WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.759242185 (45h13m50.230452925s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.774028982 (45h13m50.245239711s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.786684680 (45h13m50.257895419s) [noise=true , address=45.91.116.9:4272 , piece_id=4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.790669208 (45h13m50.261879947s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.850747877 (45h13m50.321958617s) [noise=true , address=45.91.116.8:4071 , piece_id=3YI67CEXFW75VRROYLSABR55DNVFHE4SR6EUP57CPRIIB63GZOVA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884817071 (45h13m50.3560278s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884838361 (45h13m50.35604909s) [seg_pos={0 0}, upload=39772]: Segment reached optimal threshold of 14 pieces. Dec 5 14:09:13.885054584 (45h13m50.356265313s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3): protocol: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection Dec 5 14:09:13.885064302 (45h13m50.356275041s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885416693 (45h13m50.356627432s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection Dec 5 14:09:13.885427744 (45h13m50.356638483s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885527824 (45h13m50.356738563s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection Dec 5 14:09:13.885535649 (45h13m50.356746388s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885614951 (45h13m50.35682569s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection Dec 5 14:09:13.885620010 (45h13m50.356830749s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885636241 (45h13m50.35684697s) [seg_pos={0 0}, upload=39772]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 5 14:09:13.885731973 (45h13m50.356942722s) [upload=39772]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 14:09:13.885739618 (45h13m50.356950357s) [upload=39772]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:09:14.639752295 (45h13m51.110963024s) [upload=39772]: Done uploading segments... Dec 5 14:09:14.639788193 (45h13m51.110998932s) [upload=39772]: Upload finished. err: <nil> Duration: 9.209462315s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.723865191s trace: https://jaeger.dev.storj.io/trace/41b9575307efc1e3 FAIL (2025-12-05 14:12:46 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.723931768s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X..X.......X...............................] duration: 17.995884509s trace: https://jaeger.dev.storj.io/trace/5cac00e234ac2601 FAIL (2025-12-05 14:12:46 UTC) orbiter warning: context deadline exceeded Duration: 17.995912882s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [XXXX.XXXX.XXX.XXXXXXXXXXXXXXXXXXXXXX.X..XXXXX..XXXXXX.X.XXXX] duration: 5.00107814s trace: https://jaeger.dev.storj.io/trace/5efff6e785241c78 FAIL (2025-12-05 14:12:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001140767s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 9.460071124s trace: https://jaeger.dev.storj.io/trace/65df260343fa2062 FAIL (2025-12-05 14:12:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded ======LOGS====== Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments... Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment... Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment. Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments... Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces. Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments... Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil> Duration: 9.460106352s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 3.034248381s trace: https://jaeger.dev.storj.io/trace/41b9575307efc1e3 FAIL (2025-12-05 14:12:28 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.034269272s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 12.317191154s trace: https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6 FAIL (2025-12-05 14:12:06 UTC) orbiter warning: context deadline exceeded Duration: 12.317733658s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 9.248192822s trace: https://jaeger.dev.storj.io/trace/572c91a59c970da7 FAIL (2025-12-05 14:12:00 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 9.248231977s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.93345867s trace: https://jaeger.dev.storj.io/trace/361a2ea17a62744e FAIL (2025-12-05 14:11:59 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 5 14:08:02.649750072 (45h12m39.120960811s) [upload=39762]: Starting upload Dec 5 14:08:02.650012922 (45h12m39.121223681s) [upload=39762]: Done starting upload Dec 5 14:08:02.650140385 (45h12m39.121351124s) [upload=39762]: Uploading segments... Dec 5 14:08:02.650204036 (45h12m39.121414775s) [upload=39762]: Got next segment. Inline: true Dec 5 14:08:02.650210659 (45h12m39.121421398s) [upload=39762]: Waiting for error group managing segments... Dec 5 14:08:02.650285611 (45h12m39.12149635s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 14:08:02.650292053 (45h12m39.121502792s) [upload=39762]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 5 14:08:02.650295811 (45h12m39.12150655s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:08:05.223576463 (45h12m41.694787212s) [upload=39762]: Done uploading segments... Dec 5 14:08:05.223604205 (45h12m41.694814934s) [upload=39762]: Upload finished. err: <nil> Duration:* 2.933487815s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 4.41075398s trace: https://jaeger.dev.storj.io/trace/361a2ea17a62744e FAIL (2025-12-05 14:11:44 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 5 14:08:02.649750072 (45h12m39.120960811s) [upload=39762]: Starting upload Dec 5 14:08:02.650012922 (45h12m39.121223681s) [upload=39762]: Done starting upload Dec 5 14:08:02.650140385 (45h12m39.121351124s) [upload=39762]: Uploading segments... Dec 5 14:08:02.650204036 (45h12m39.121414775s) [upload=39762]: Got next segment. Inline: true Dec 5 14:08:02.650210659 (45h12m39.121421398s) [upload=39762]: Waiting for error group managing segments... Dec 5 14:08:02.650285611 (45h12m39.12149635s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 14:08:02.650292053 (45h12m39.121502792s) [upload=39762]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 5 14:08:02.650295811 (45h12m39.12150655s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:08:05.223576463 (45h12m41.694787212s) [upload=39762]: Done uploading segments... Dec 5 14:08:05.223604205 (45h12m41.694814934s) [upload=39762]: Upload finished. err: <nil> Duration:* 4.410785329s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 14.855436569s trace: https://jaeger.dev.storj.io/trace/65df260343fa2062 FAIL (2025-12-05 14:11:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded ======LOGS====== Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments... Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment... Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment. Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments... Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces. Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments... Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil> Duration: 14.855467847s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X..X.......X..............................] duration: 15.630804821s trace: https://jaeger.dev.storj.io/trace/5cac00e234ac2601 FAIL (2025-12-05 14:11:42 UTC) orbiter warning: context deadline exceeded Duration: 15.630820019s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 8.156344637s trace: https://jaeger.dev.storj.io/trace/70e54e58d52fa95e FAIL (2025-12-05 14:11:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, Piece ID: MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q) context deadline exceeded; piecestore: (Node ID: 12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, Piece ID: UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ) context deadline exceeded; piecestore: (Node ID: 12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4, Piece ID: 4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA) context deadline exceeded; piecestore: (Node ID: 12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, Piece ID: SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA) context deadline exceeded; piecestore: (Node ID: 123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS, Piece ID: MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA) context deadline exceeded; piecestore: (Node ID: 121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, Piece ID: EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ) context deadline exceeded; piecestore: (Node ID: 1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY, Piece ID: CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA) context deadline exceeded ======LOGS====== 4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669180895 (45h13m48.140391634s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669641381 (45h13m48.14085212s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670111447 (45h13m48.141322196s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670601259 (45h13m48.141811999s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671075502 (45h13m48.142286241s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671598037 (45h13m48.142808776s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672127346 (45h13m48.143338105s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672648588 (45h13m48.143859327s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673155192 (45h13m48.144365931s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673691684 (45h13m48.144902423s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.674805966 (45h13m48.146016705s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.675360772 (45h13m48.146571511s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:13.239845356 (45h13m49.711056085s) [upload=39772]: Next returned nil segment Dec 5 14:09:13.239860604 (45h13m49.711071334s) [upload=39772]: Waiting for error group managing segments... Dec 5 14:09:13.643777330 (45h13m50.114988069s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.697999800 (45h13m50.169210539s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.713127946 (45h13m50.184338685s) [noise=true , address=77.90.31.15:4729 , piece_id=IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.718498752 (45h13m50.189709491s) [noise=true , address=77.90.31.21:4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.726522451 (45h13m50.19773318s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.749057442 (45h13m50.220268171s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.750808776 (45h13m50.222019515s) [noise=true , address=77.90.31.10:4542 , piece_id=XDD4QZAKCWJK5A3F5FDL4NRNPEQZF3GKODCGVT2V47AH6KVEOOWQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.754762756 (45h13m50.225973485s) [noise=true , address=77.90.31.11:4556 , piece_id=WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.759242185 (45h13m50.230452925s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.774028982 (45h13m50.245239711s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.786684680 (45h13m50.257895419s) [noise=true , address=45.91.116.9:4272 , piece_id=4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.790669208 (45h13m50.261879947s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.850747877 (45h13m50.321958617s) [noise=true , address=45.91.116.8:4071 , piece_id=3YI67CEXFW75VRROYLSABR55DNVFHE4SR6EUP57CPRIIB63GZOVA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884817071 (45h13m50.3560278s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884838361 (45h13m50.35604909s) [seg_pos={0 0}, upload=39772]: Segment reached optimal threshold of 14 pieces. Dec 5 14:09:13.885054584 (45h13m50.356265313s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3): protocol: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection Dec 5 14:09:13.885064302 (45h13m50.356275041s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885416693 (45h13m50.356627432s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection Dec 5 14:09:13.885427744 (45h13m50.356638483s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885527824 (45h13m50.356738563s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection Dec 5 14:09:13.885535649 (45h13m50.356746388s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885614951 (45h13m50.35682569s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection Dec 5 14:09:13.885620010 (45h13m50.356830749s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885636241 (45h13m50.35684697s) [seg_pos={0 0}, upload=39772]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 5 14:09:13.885731973 (45h13m50.356942722s) [upload=39772]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 14:09:13.885739618 (45h13m50.356950357s) [upload=39772]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:09:14.639752295 (45h13m51.110963024s) [upload=39772]: Done uploading segments... Dec 5 14:09:14.639788193 (45h13m51.110998932s) [upload=39772]: Upload finished. err: <nil> Duration: 8.156374424s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [.XXXX.XXXX.XXX.XXXXXXXXXXXXXXXXXXXXXX.X..XXXXX..XXXXXX.X.XXX] duration: 5.001170485s trace: https://jaeger.dev.storj.io/trace/44170f13e634c243 FAIL (2025-12-05 14:11:39 UTC) orbiter warning: context deadline exceeded Duration: 5.001195101s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.865113189s trace: https://jaeger.dev.storj.io/trace/41b9575307efc1e3 FAIL (2025-12-05 14:11:38 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.865138177s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 12.721972122s trace: https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6 FAIL (2025-12-05 14:11:32 UTC) orbiter warning: context deadline exceeded Duration: 12.72200233s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X..X.......X.............................] duration: 21.266292066s trace: https://jaeger.dev.storj.io/trace/5cac00e234ac2601 FAIL (2025-12-05 14:11:27 UTC) orbiter warning: context deadline exceeded Duration: 21.266356338s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 3.809949419s trace: https://jaeger.dev.storj.io/trace/361a2ea17a62744e FAIL (2025-12-05 14:11:13 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 5 14:08:02.649750072 (45h12m39.120960811s) [upload=39762]: Starting upload Dec 5 14:08:02.650012922 (45h12m39.121223681s) [upload=39762]: Done starting upload Dec 5 14:08:02.650140385 (45h12m39.121351124s) [upload=39762]: Uploading segments... Dec 5 14:08:02.650204036 (45h12m39.121414775s) [upload=39762]: Got next segment. Inline: true Dec 5 14:08:02.650210659 (45h12m39.121421398s) [upload=39762]: Waiting for error group managing segments... Dec 5 14:08:02.650285611 (45h12m39.12149635s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 14:08:02.650292053 (45h12m39.121502792s) [upload=39762]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 5 14:08:02.650295811 (45h12m39.12150655s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:08:05.223576463 (45h12m41.694787212s) [upload=39762]: Done uploading segments... Dec 5 14:08:05.223604205 (45h12m41.694814934s) [upload=39762]: Upload finished. err: <nil> Duration:* 3.809980168s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 9.707319869s trace: https://jaeger.dev.storj.io/trace/65df260343fa2062 FAIL (2025-12-05 14:11:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded ======LOGS====== Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments... Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment... Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment. Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments... Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces. Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments... Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil> Duration: 9.707348964s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X..X.......X............................] duration: 18.952106825s trace: https://jaeger.dev.storj.io/trace/5cac00e234ac2601 FAIL (2025-12-05 14:11:05 UTC) orbiter warning: context deadline exceeded Duration: 18.952126052s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 12.289952718s trace: https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6 FAIL (2025-12-05 14:11:04 UTC) orbiter warning: context deadline exceeded Duration: 12.28999563s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 7.352351631s trace: https://jaeger.dev.storj.io/trace/572c91a59c970da7 FAIL (2025-12-05 14:10:58 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 7.352381027s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [..XXXX.XXXX.XXX.XXXXXXXXXXXXXXXXXXXXXX.X..XXXXX..XXXXXX.X.XX] duration: 5.002530796s trace: https://jaeger.dev.storj.io/trace/1ebe9dff23b91456 FAIL (2025-12-05 14:10:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002552917s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X..X.......X...........................] duration: 26.008901257s trace: https://jaeger.dev.storj.io/trace/5cac00e234ac2601 FAIL (2025-12-05 14:10:46 UTC) orbiter warning: context deadline exceeded Duration: 26.008944491s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 9.419931876s trace: https://jaeger.dev.storj.io/trace/65df260343fa2062 FAIL (2025-12-05 14:10:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded ======LOGS====== Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments... Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment... Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment. Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments... Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces. Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments... Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil> Duration: 9.41996054s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.90892202s trace: https://jaeger.dev.storj.io/trace/41b9575307efc1e3 FAIL (2025-12-05 14:10:32 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.908950454s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 8.007464257s trace: https://jaeger.dev.storj.io/trace/70e54e58d52fa95e FAIL (2025-12-05 14:10:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, Piece ID: MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q) context deadline exceeded; piecestore: (Node ID: 12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, Piece ID: UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ) context deadline exceeded; piecestore: (Node ID: 12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4, Piece ID: 4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA) context deadline exceeded; piecestore: (Node ID: 12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, Piece ID: SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA) context deadline exceeded; piecestore: (Node ID: 123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS, Piece ID: MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA) context deadline exceeded; piecestore: (Node ID: 121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, Piece ID: EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ) context deadline exceeded; piecestore: (Node ID: 1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY, Piece ID: CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA) context deadline exceeded ======LOGS====== 4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669180895 (45h13m48.140391634s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669641381 (45h13m48.14085212s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670111447 (45h13m48.141322196s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670601259 (45h13m48.141811999s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671075502 (45h13m48.142286241s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671598037 (45h13m48.142808776s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672127346 (45h13m48.143338105s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672648588 (45h13m48.143859327s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673155192 (45h13m48.144365931s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673691684 (45h13m48.144902423s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.674805966 (45h13m48.146016705s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.675360772 (45h13m48.146571511s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:13.239845356 (45h13m49.711056085s) [upload=39772]: Next returned nil segment Dec 5 14:09:13.239860604 (45h13m49.711071334s) [upload=39772]: Waiting for error group managing segments... Dec 5 14:09:13.643777330 (45h13m50.114988069s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.697999800 (45h13m50.169210539s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.713127946 (45h13m50.184338685s) [noise=true , address=77.90.31.15:4729 , piece_id=IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.718498752 (45h13m50.189709491s) [noise=true , address=77.90.31.21:4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.726522451 (45h13m50.19773318s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.749057442 (45h13m50.220268171s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.750808776 (45h13m50.222019515s) [noise=true , address=77.90.31.10:4542 , piece_id=XDD4QZAKCWJK5A3F5FDL4NRNPEQZF3GKODCGVT2V47AH6KVEOOWQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.754762756 (45h13m50.225973485s) [noise=true , address=77.90.31.11:4556 , piece_id=WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.759242185 (45h13m50.230452925s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.774028982 (45h13m50.245239711s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.786684680 (45h13m50.257895419s) [noise=true , address=45.91.116.9:4272 , piece_id=4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.790669208 (45h13m50.261879947s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.850747877 (45h13m50.321958617s) [noise=true , address=45.91.116.8:4071 , piece_id=3YI67CEXFW75VRROYLSABR55DNVFHE4SR6EUP57CPRIIB63GZOVA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884817071 (45h13m50.3560278s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884838361 (45h13m50.35604909s) [seg_pos={0 0}, upload=39772]: Segment reached optimal threshold of 14 pieces. Dec 5 14:09:13.885054584 (45h13m50.356265313s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3): protocol: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection Dec 5 14:09:13.885064302 (45h13m50.356275041s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885416693 (45h13m50.356627432s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection Dec 5 14:09:13.885427744 (45h13m50.356638483s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885527824 (45h13m50.356738563s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection Dec 5 14:09:13.885535649 (45h13m50.356746388s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885614951 (45h13m50.35682569s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection Dec 5 14:09:13.885620010 (45h13m50.356830749s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885636241 (45h13m50.35684697s) [seg_pos={0 0}, upload=39772]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 5 14:09:13.885731973 (45h13m50.356942722s) [upload=39772]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 14:09:13.885739618 (45h13m50.356950357s) [upload=39772]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:09:14.639752295 (45h13m51.110963024s) [upload=39772]: Done uploading segments... Dec 5 14:09:14.639788193 (45h13m51.110998932s) [upload=39772]: Upload finished. err: <nil> Duration: 8.007496277s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [X..XXXX.XXXX.XXX.XXXXXXXXXXXXXXXXXXXXXX.X..XXXXX..XXXXXX.X.X] duration: 5.001593439s trace: https://jaeger.dev.storj.io/trace/5d3bf7c6cf0bc939 FAIL (2025-12-05 14:10:25 UTC) orbiter warning: context deadline exceeded Duration: 5.001617416s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.981453212s trace: https://jaeger.dev.storj.io/trace/361a2ea17a62744e FAIL (2025-12-05 14:10:19 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 5 14:08:02.649750072 (45h12m39.120960811s) [upload=39762]: Starting upload Dec 5 14:08:02.650012922 (45h12m39.121223681s) [upload=39762]: Done starting upload Dec 5 14:08:02.650140385 (45h12m39.121351124s) [upload=39762]: Uploading segments... Dec 5 14:08:02.650204036 (45h12m39.121414775s) [upload=39762]: Got next segment. Inline: true Dec 5 14:08:02.650210659 (45h12m39.121421398s) [upload=39762]: Waiting for error group managing segments... Dec 5 14:08:02.650285611 (45h12m39.12149635s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 14:08:02.650292053 (45h12m39.121502792s) [upload=39762]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 5 14:08:02.650295811 (45h12m39.12150655s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:08:05.223576463 (45h12m41.694787212s) [upload=39762]: Done uploading segments... Dec 5 14:08:05.223604205 (45h12m41.694814934s) [upload=39762]: Upload finished. err: <nil> Duration:* 2.981487787s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 8.796632385s trace: https://jaeger.dev.storj.io/trace/65df260343fa2062 FAIL (2025-12-05 14:10:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded ======LOGS====== Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments... Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment... Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment. Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments... Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces. Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments... Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil> Duration: 8.796686547s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 17.76972987s trace: https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6 FAIL (2025-12-05 14:10:08 UTC) orbiter warning: context deadline exceeded Duration: 17.769785386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 4.095778626s trace: https://jaeger.dev.storj.io/trace/41b9575307efc1e3 FAIL (2025-12-05 14:10:07 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.095816539s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 8.624385618s trace: https://jaeger.dev.storj.io/trace/70e54e58d52fa95e FAIL (2025-12-05 14:10:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, Piece ID: MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q) context deadline exceeded; piecestore: (Node ID: 12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, Piece ID: UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ) context deadline exceeded; piecestore: (Node ID: 12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4, Piece ID: 4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA) context deadline exceeded; piecestore: (Node ID: 12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, Piece ID: SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA) context deadline exceeded; piecestore: (Node ID: 123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS, Piece ID: MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA) context deadline exceeded; piecestore: (Node ID: 121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, Piece ID: EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ) context deadline exceeded; piecestore: (Node ID: 1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY, Piece ID: CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA) context deadline exceeded ======LOGS====== 4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669180895 (45h13m48.140391634s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.669641381 (45h13m48.14085212s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670111447 (45h13m48.141322196s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.670601259 (45h13m48.141811999s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671075502 (45h13m48.142286241s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.671598037 (45h13m48.142808776s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672127346 (45h13m48.143338105s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.672648588 (45h13m48.143859327s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673155192 (45h13m48.144365931s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.673691684 (45h13m48.144902423s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.674805966 (45h13m48.146016705s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:11.675360772 (45h13m48.146571511s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Uploading piece... Dec 5 14:09:13.239845356 (45h13m49.711056085s) [upload=39772]: Next returned nil segment Dec 5 14:09:13.239860604 (45h13m49.711071334s) [upload=39772]: Waiting for error group managing segments... Dec 5 14:09:13.643777330 (45h13m50.114988069s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.697999800 (45h13m50.169210539s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.713127946 (45h13m50.184338685s) [noise=true , address=77.90.31.15:4729 , piece_id=IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.718498752 (45h13m50.189709491s) [noise=true , address=77.90.31.21:4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.726522451 (45h13m50.19773318s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.749057442 (45h13m50.220268171s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.750808776 (45h13m50.222019515s) [noise=true , address=77.90.31.10:4542 , piece_id=XDD4QZAKCWJK5A3F5FDL4NRNPEQZF3GKODCGVT2V47AH6KVEOOWQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.754762756 (45h13m50.225973485s) [noise=true , address=77.90.31.11:4556 , piece_id=WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.759242185 (45h13m50.230452925s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.774028982 (45h13m50.245239711s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.786684680 (45h13m50.257895419s) [noise=true , address=45.91.116.9:4272 , piece_id=4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.790669208 (45h13m50.261879947s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.850747877 (45h13m50.321958617s) [noise=true , address=45.91.116.8:4071 , piece_id=3YI67CEXFW75VRROYLSABR55DNVFHE4SR6EUP57CPRIIB63GZOVA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884817071 (45h13m50.3560278s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil> Dec 5 14:09:13.884838361 (45h13m50.35604909s) [seg_pos={0 0}, upload=39772]: Segment reached optimal threshold of 14 pieces. Dec 5 14:09:13.885054584 (45h13m50.356265313s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3): protocol: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection Dec 5 14:09:13.885064302 (45h13m50.356275041s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885416693 (45h13m50.356627432s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection Dec 5 14:09:13.885427744 (45h13m50.356638483s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885527824 (45h13m50.356738563s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection Dec 5 14:09:13.885535649 (45h13m50.356746388s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885614951 (45h13m50.35682569s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection Dec 5 14:09:13.885620010 (45h13m50.356830749s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold Dec 5 14:09:13.885636241 (45h13m50.35684697s) [seg_pos={0 0}, upload=39772]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 5 14:09:13.885731973 (45h13m50.356942722s) [upload=39772]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 14:09:13.885739618 (45h13m50.356950357s) [upload=39772]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:09:14.639752295 (45h13m51.110963024s) [upload=39772]: Done uploading segments... Dec 5 14:09:14.639788193 (45h13m51.110998932s) [upload=39772]: Upload finished. err: <nil> Duration: 8.624415555s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 9.035895685s trace: https://jaeger.dev.storj.io/trace/572c91a59c970da7 FAIL (2025-12-05 14:10:03 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 9.035926353s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X..X.......X..........................] duration: 22.790369359s trace: https://jaeger.dev.storj.io/trace/5cac00e234ac2601 FAIL (2025-12-05 14:09:59 UTC) orbiter warning: context deadline exceeded Duration: 22.790394838s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 10.174597106s trace: https://jaeger.dev.storj.io/trace/65df260343fa2062 FAIL (2025-12-05 14:09:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded ======LOGS====== Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments... Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment... Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment. Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece... Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments... Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil> Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces. Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments... Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil> Duration: 10.174624818s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.972250578s trace: https://jaeger.dev.storj.io/trace/361a2ea17a62744e FAIL (2025-12-05 14:09:40 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 5 14:08:02.649750072 (45h12m39.120960811s) [upload=39762]: Starting upload Dec 5 14:08:02.650012922 (45h12m39.121223681s) [upload=39762]: Done starting upload Dec 5 14:08:02.650140385 (45h12m39.121351124s) [upload=39762]: Uploading segments... Dec 5 14:08:02.650204036 (45h12m39.121414775s) [upload=39762]: Got next segment. Inline: true Dec 5 14:08:02.650210659 (45h12m39.121421398s) [upload=39762]: Waiting for error group managing segments... Dec 5 14:08:02.650285611 (45h12m39.12149635s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 14:08:02.650292053 (45h12m39.121502792s) [upload=39762]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 5 14:08:02.650295811 (45h12m39.12150655s) [upload=39762]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 14:08:05.223576463 (45h12m41.694787212s) [upload=39762]: Done uploading segments... Dec 5 14:08:05.223604205 (45h12m41.694814934s) [upload=39762]: Upload finished. err: <nil> Duration:* 2.972276157s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)