Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 5.001540248s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:45:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00156238s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.416064976s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:44:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.416092399s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.728899062s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:44:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.728925603s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X.........X...........X...............X] duration: 30.003399582s trace: https://jaeger.dev.storj.io/trace/3eff27cd9e59e739 FAIL (2025-12-25 09:44:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: HFA4RRO2ILMUYBWPE5OOW2ARK3YTA6M6YQDMHPM55ECH5YPLXAUA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: IKYKESLO7UCB4BRMMGYBSGNE6MF73OAD6LUIAAXUR5VLLUCGLXWA) context deadline exceeded ======LOGS====== Dec 25 09:44:07.264250362 (474h15m7.135333407s) [upload=416292]: Starting upload Dec 25 09:44:07.264302622 (474h15m7.135385657s) [upload=416292]: Done starting upload Dec 25 09:44:07.264491972 (474h15m7.135575007s) [upload=416292]: Uploading segments... Dec 25 09:44:07.264540494 (474h15m7.135623529s) [upload=416292]: Got next segment. Inline: false Dec 25 09:44:07.264563668 (474h15m7.135646703s) [upload=416292]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:44:07.264568638 (474h15m7.135651673s) [upload=416292]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:44:07.508329921 (474h15m7.379412966s) [seg_pos={0 0}, upload=416292]: Begin upload segment... Dec 25 09:44:07.508371269 (474h15m7.379454304s) [seg_pos={0 0}, upload=416292]: Done begin upload segment. Dec 25 09:44:07.508395897 (474h15m7.379478932s) [noise=true , address=145.79.227.11:4729 , piece_id=E527Q3N43SKXLCIWASMKTQZ5WVNYKITNHNZUUXWARVSMHZPOAVSA, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.508532487 (474h15m7.379615522s) [noise=true , address=145.79.227.17:4971 , piece_id=AK7OC6NMAZAXP2CA2266AM4BDWDADJ73VLPTDIXYWQ2K4N32GT2A, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.509394097 (474h15m7.380477142s) [noise=true , address=145.79.227.45:4230 , piece_id=QI3QODZPMFQGWZ7CFGJYPDVLS5MSTDL6O3VTIUTF4KEBN2DTQTNA, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.509856548 (474h15m7.380939583s) [noise=true , address=145.79.227.29:4942 , piece_id=MVUTYJNMHSNZOMBCAWVV65AIAQD7C3NWKRPJ2EOPLLXLMMR5EKXQ, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.510410062 (474h15m7.381493097s) [noise=true , address=145.79.227.52:4447 , piece_id=5YBWBH72YFTTTICIEWFCREFN5GLSMYOKWGCM3LBLMMA7ZUCRKN2A, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.510884334 (474h15m7.38196738s) [noise=true , address=145.79.227.41:4087 , piece_id=KOQTJX5FLYBAVT5XZSQTZ6EAUDXNWC2A3WZPZTZFFCAA3TC7H6QA, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.511552066 (474h15m7.382635111s) [noise=true , address=145.79.227.38:4979 , piece_id=C3UTDMVX22JDERB36UGH46TMI2WBKEBP56SUT5Q7KBO4URNRZSMA, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.511760623 (474h15m7.382843658s) [noise=true , address=45.91.116.11:4120 , piece_id=HFA4RRO2ILMUYBWPE5OOW2ARK3YTA6M6YQDMHPM55ECH5YPLXAUA, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.512685765 (474h15m7.38376881s) [noise=true , address=145.79.227.46:4601 , piece_id=VIKTFGPXOP35APA2UCB7H3MVLIT6BDZ52MM52MVYU3PHIEAZLRPA, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.512793029 (474h15m7.383876074s) [noise=true , address=145.79.227.39:4015 , piece_id=C5OB5NN662K6JHYXZDYCQHQWVVD64YLGYZWXL4WZG7JWAGCHTASQ, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.513667013 (474h15m7.384750048s) [noise=true , address=145.79.227.40:4049 , piece_id=T5ZOIDVERBPRVCO2U2YOUSKOTGSKCNXIEDR7QC2NZJ7GRFVJKE6A, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.514128802 (474h15m7.385211837s) [noise=true , address=145.79.227.2:4113 , piece_id=BG7ABTYXGZDPWRSQ56RJLIUWIYNHJE7HADVTKYBMUGNUPWJP3ZGA, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.514196191 (474h15m7.385279236s) [noise=true , address=77.90.31.14:4661 , piece_id=MKFZ7SAXQO7EX2GYZM5XIPVVIPZNVCTOVF4MDAV7U7SPN5NSENTQ, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.514614277 (474h15m7.385697322s) [noise=true , address=77.90.31.11:4556 , piece_id=IKYKESLO7UCB4BRMMGYBSGNE6MF73OAD6LUIAAXUR5VLLUCGLXWA, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.514936030 (474h15m7.386019065s) [noise=true , address=45.91.116.8:4047 , piece_id=3UNU3PA25FU4IJR2JK3TFMU4QW3FE2DXJ6QB4PF7OAXLAFJ4LG6Q, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.515285866 (474h15m7.386368901s) [noise=true , address=45.91.116.6:4974 , piece_id=LFVW3SICWO5U22K3WCSKWTOMLP2JKZBL3E4XOZRONKBQYRRON5MQ, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:09.134358899 (474h15m9.005441944s) [upload=416292]: Next returned nil segment Dec 25 09:44:09.134379950 (474h15m9.005463025s) [upload=416292]: Waiting for error group managing segments... Dec 25 09:44:09.516963432 (474h15m9.388046477s) [noise=true , address=145.79.227.11:4729 , piece_id=E527Q3N43SKXLCIWASMKTQZ5WVNYKITNHNZUUXWARVSMHZPOAVSA, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.525132447 (474h15m9.396215492s) [noise=true , address=145.79.227.17:4971 , piece_id=AK7OC6NMAZAXP2CA2266AM4BDWDADJ73VLPTDIXYWQ2K4N32GT2A, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.525291330 (474h15m9.396374365s) [noise=true , address=145.79.227.38:4979 , piece_id=C3UTDMVX22JDERB36UGH46TMI2WBKEBP56SUT5Q7KBO4URNRZSMA, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.525644662 (474h15m9.396727697s) [noise=true , address=145.79.227.29:4942 , piece_id=MVUTYJNMHSNZOMBCAWVV65AIAQD7C3NWKRPJ2EOPLLXLMMR5EKXQ, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.526018924 (474h15m9.397101969s) [noise=true , address=145.79.227.46:4601 , piece_id=VIKTFGPXOP35APA2UCB7H3MVLIT6BDZ52MM52MVYU3PHIEAZLRPA, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.526269031 (474h15m9.397352066s) [noise=true , address=145.79.227.39:4015 , piece_id=C5OB5NN662K6JHYXZDYCQHQWVVD64YLGYZWXL4WZG7JWAGCHTASQ, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.526944357 (474h15m9.398027392s) [noise=true , address=145.79.227.2:4113 , piece_id=BG7ABTYXGZDPWRSQ56RJLIUWIYNHJE7HADVTKYBMUGNUPWJP3ZGA, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.530206759 (474h15m9.401289804s) [noise=true , address=145.79.227.45:4230 , piece_id=QI3QODZPMFQGWZ7CFGJYPDVLS5MSTDL6O3VTIUTF4KEBN2DTQTNA, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.538734367 (474h15m9.409817412s) [noise=true , address=145.79.227.41:4087 , piece_id=KOQTJX5FLYBAVT5XZSQTZ6EAUDXNWC2A3WZPZTZFFCAA3TC7H6QA, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.544202299 (474h15m9.415285344s) [noise=true , address=145.79.227.52:4447 , piece_id=5YBWBH72YFTTTICIEWFCREFN5GLSMYOKWGCM3LBLMMA7ZUCRKN2A, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.731088561 (474h15m9.602171606s) [noise=true , address=77.90.31.14:4661 , piece_id=MKFZ7SAXQO7EX2GYZM5XIPVVIPZNVCTOVF4MDAV7U7SPN5NSENTQ, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.772766583 (474h15m9.643849628s) [noise=true , address=45.91.116.11:4120 , piece_id=HFA4RRO2ILMUYBWPE5OOW2ARK3YTA6M6YQDMHPM55ECH5YPLXAUA, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:10.165733478 (474h15m10.036816523s) [noise=true , address=77.90.31.11:4556 , piece_id=IKYKESLO7UCB4BRMMGYBSGNE6MF73OAD6LUIAAXUR5VLLUCGLXWA, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:10.165760330 (474h15m10.036843365s) [seg_pos={0 0}, upload=416292]: Segment reached optimal threshold of 13 pieces. Dec 25 09:44:10.166041665 (474h15m10.0371247s) [noise=true , address=45.91.116.6:4974 , piece_id=LFVW3SICWO5U22K3WCSKWTOMLP2JKZBL3E4XOZRONKBQYRRON5MQ, seg_pos={0 0}, upload=416292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 25 09:44:10.166051814 (474h15m10.037134859s) [noise=true , address=45.91.116.6:4974 , piece_id=LFVW3SICWO5U22K3WCSKWTOMLP2JKZBL3E4XOZRONKBQYRRON5MQ, seg_pos={0 0}, upload=416292]: Long tail cancellation due to optimal threshold Dec 25 09:44:10.166225455 (474h15m10.03730851s) [noise=true , address=45.91.116.8:4047 , piece_id=3UNU3PA25FU4IJR2JK3TFMU4QW3FE2DXJ6QB4PF7OAXLAFJ4LG6Q, seg_pos={0 0}, upload=416292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:34726->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:34726->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34726->45.91.116.8:4047: use of closed network connection Dec 25 09:44:10.166237518 (474h15m10.037320563s) [noise=true , address=45.91.116.8:4047 , piece_id=3UNU3PA25FU4IJR2JK3TFMU4QW3FE2DXJ6QB4PF7OAXLAFJ4LG6Q, seg_pos={0 0}, upload=416292]: Long tail cancellation due to optimal threshold Dec 25 09:44:10.166348600 (474h15m10.037431645s) [noise=true , address=145.79.227.40:4049 , piece_id=T5ZOIDVERBPRVCO2U2YOUSKOTGSKCNXIEDR7QC2NZJ7GRFVJKE6A, seg_pos={0 0}, upload=416292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 09:44:10.166358769 (474h15m10.037441814s) [noise=true , address=145.79.227.40:4049 , piece_id=T5ZOIDVERBPRVCO2U2YOUSKOTGSKCNXIEDR7QC2NZJ7GRFVJKE6A, seg_pos={0 0}, upload=416292]: Long tail cancellation due to optimal threshold Dec 25 09:44:10.166380861 (474h15m10.037463907s) [seg_pos={0 0}, upload=416292]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:44:10.166466955 (474h15m10.03755s) [upload=416292]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:44:10.166474319 (474h15m10.037557364s) [upload=416292]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:44:10.450581867 (474h15m10.321664922s) [upload=416292]: Done uploading segments... Dec 25 09:44:10.450603749 (474h15m10.321686784s) [upload=416292]: Upload finished. err: <nil> Duration: 30.003414651s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 4.072158819s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:43:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.072184579s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.........X...........X...............] duration: 5.731392313s trace: https://jaeger.dev.storj.io/trace/795e298d20174ea1 FAIL (2025-12-25 09:43:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ]: Done starting upload Dec 25 09:33:01.262724874 (474h4m1.133807919s) [upload=416129]: Uploading segments... Dec 25 09:33:01.262992493 (474h4m1.134075538s) [upload=416129]: Got next segment. Inline: false Dec 25 09:33:01.263043731 (474h4m1.134126776s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:33:01.263050604 (474h4m1.134133649s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:33:01.511625191 (474h4m1.382708226s) [seg_pos={0 0}, upload=416129]: Begin upload segment... Dec 25 09:33:01.511677211 (474h4m1.382760246s) [seg_pos={0 0}, upload=416129]: Done begin upload segment. Dec 25 09:33:01.511706086 (474h4m1.382789131s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.520762050 (474h4m1.391845095s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.522962850 (474h4m1.394045895s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.528425993 (474h4m1.399509038s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.531387723 (474h4m1.402470768s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.532761669 (474h4m1.403844715s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535007145 (474h4m1.40609019s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535436131 (474h4m1.406519176s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.536867767 (474h4m1.407950812s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.537459183 (474h4m1.408542228s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.538526586 (474h4m1.409609631s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.540585998 (474h4m1.411669043s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542237873 (474h4m1.413320948s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542542713 (474h4m1.413625748s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.545729362 (474h4m1.416812407s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.546027639 (474h4m1.417110684s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:03.159768330 (474h4m3.030851375s) [upload=416129]: Next returned nil segment Dec 25 09:33:03.159785773 (474h4m3.030868808s) [upload=416129]: Waiting for error group managing segments... Dec 25 09:33:03.547727689 (474h4m3.418810734s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.548444835 (474h4m3.41952789s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.551962984 (474h4m3.423046039s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.560368159 (474h4m3.431451204s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.562870935 (474h4m3.43395398s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.563062640 (474h4m3.434145705s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.573975500 (474h4m3.445058545s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.582677770 (474h4m3.453760815s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.588464799 (474h4m3.459547844s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.712544699 (474h4m3.583627744s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.851482769 (474h4m3.722565814s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.882255556 (474h4m3.753338601s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918839348 (474h4m3.789922403s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918864156 (474h4m3.789947201s) [seg_pos={0 0}, upload=416129]: Segment reached optimal threshold of 13 pieces. Dec 25 09:33:03.919269627 (474h4m3.790352672s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 09:33:03.919280899 (474h4m3.790363944s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919532328 (474h4m3.790615373s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection Dec 25 09:33:03.919541034 (474h4m3.790624079s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919588836 (474h4m3.790671871s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection Dec 25 09:33:03.919609945 (474h4m3.79069299s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919639131 (474h4m3.790722176s) [seg_pos={0 0}, upload=416129]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:33:03.919766364 (474h4m3.790849409s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:33:03.919775921 (474h4m3.790858977s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:33:04.215163028 (474h4m4.086246073s) [upload=416129]: Done uploading segments... Dec 25 09:33:04.215194258 (474h4m4.086277303s) [upload=416129]: Upload finished. err: <nil> Duration: 5.731426689s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.644304594s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:43:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.644331555s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.441899972s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:42:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.441926592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X......................X.........X...........X..............] duration: 5.700524518s trace: https://jaeger.dev.storj.io/trace/795e298d20174ea1 FAIL (2025-12-25 09:42:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ]: Done starting upload Dec 25 09:33:01.262724874 (474h4m1.133807919s) [upload=416129]: Uploading segments... Dec 25 09:33:01.262992493 (474h4m1.134075538s) [upload=416129]: Got next segment. Inline: false Dec 25 09:33:01.263043731 (474h4m1.134126776s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:33:01.263050604 (474h4m1.134133649s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:33:01.511625191 (474h4m1.382708226s) [seg_pos={0 0}, upload=416129]: Begin upload segment... Dec 25 09:33:01.511677211 (474h4m1.382760246s) [seg_pos={0 0}, upload=416129]: Done begin upload segment. Dec 25 09:33:01.511706086 (474h4m1.382789131s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.520762050 (474h4m1.391845095s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.522962850 (474h4m1.394045895s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.528425993 (474h4m1.399509038s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.531387723 (474h4m1.402470768s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.532761669 (474h4m1.403844715s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535007145 (474h4m1.40609019s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535436131 (474h4m1.406519176s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.536867767 (474h4m1.407950812s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.537459183 (474h4m1.408542228s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.538526586 (474h4m1.409609631s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.540585998 (474h4m1.411669043s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542237873 (474h4m1.413320948s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542542713 (474h4m1.413625748s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.545729362 (474h4m1.416812407s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.546027639 (474h4m1.417110684s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:03.159768330 (474h4m3.030851375s) [upload=416129]: Next returned nil segment Dec 25 09:33:03.159785773 (474h4m3.030868808s) [upload=416129]: Waiting for error group managing segments... Dec 25 09:33:03.547727689 (474h4m3.418810734s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.548444835 (474h4m3.41952789s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.551962984 (474h4m3.423046039s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.560368159 (474h4m3.431451204s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.562870935 (474h4m3.43395398s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.563062640 (474h4m3.434145705s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.573975500 (474h4m3.445058545s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.582677770 (474h4m3.453760815s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.588464799 (474h4m3.459547844s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.712544699 (474h4m3.583627744s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.851482769 (474h4m3.722565814s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.882255556 (474h4m3.753338601s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918839348 (474h4m3.789922403s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918864156 (474h4m3.789947201s) [seg_pos={0 0}, upload=416129]: Segment reached optimal threshold of 13 pieces. Dec 25 09:33:03.919269627 (474h4m3.790352672s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 09:33:03.919280899 (474h4m3.790363944s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919532328 (474h4m3.790615373s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection Dec 25 09:33:03.919541034 (474h4m3.790624079s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919588836 (474h4m3.790671871s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection Dec 25 09:33:03.919609945 (474h4m3.79069299s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919639131 (474h4m3.790722176s) [seg_pos={0 0}, upload=416129]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:33:03.919766364 (474h4m3.790849409s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:33:03.919775921 (474h4m3.790858977s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:33:04.215163028 (474h4m4.086246073s) [upload=416129]: Done uploading segments... Dec 25 09:33:04.215194258 (474h4m4.086277303s) [upload=416129]: Upload finished. err: <nil> Duration: 5.700551811s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X......................X.........X...........X.............] duration: 5.781832662s trace: https://jaeger.dev.storj.io/trace/795e298d20174ea1 FAIL (2025-12-25 09:42:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ]: Done starting upload Dec 25 09:33:01.262724874 (474h4m1.133807919s) [upload=416129]: Uploading segments... Dec 25 09:33:01.262992493 (474h4m1.134075538s) [upload=416129]: Got next segment. Inline: false Dec 25 09:33:01.263043731 (474h4m1.134126776s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:33:01.263050604 (474h4m1.134133649s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:33:01.511625191 (474h4m1.382708226s) [seg_pos={0 0}, upload=416129]: Begin upload segment... Dec 25 09:33:01.511677211 (474h4m1.382760246s) [seg_pos={0 0}, upload=416129]: Done begin upload segment. Dec 25 09:33:01.511706086 (474h4m1.382789131s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.520762050 (474h4m1.391845095s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.522962850 (474h4m1.394045895s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.528425993 (474h4m1.399509038s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.531387723 (474h4m1.402470768s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.532761669 (474h4m1.403844715s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535007145 (474h4m1.40609019s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535436131 (474h4m1.406519176s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.536867767 (474h4m1.407950812s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.537459183 (474h4m1.408542228s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.538526586 (474h4m1.409609631s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.540585998 (474h4m1.411669043s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542237873 (474h4m1.413320948s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542542713 (474h4m1.413625748s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.545729362 (474h4m1.416812407s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.546027639 (474h4m1.417110684s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:03.159768330 (474h4m3.030851375s) [upload=416129]: Next returned nil segment Dec 25 09:33:03.159785773 (474h4m3.030868808s) [upload=416129]: Waiting for error group managing segments... Dec 25 09:33:03.547727689 (474h4m3.418810734s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.548444835 (474h4m3.41952789s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.551962984 (474h4m3.423046039s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.560368159 (474h4m3.431451204s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.562870935 (474h4m3.43395398s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.563062640 (474h4m3.434145705s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.573975500 (474h4m3.445058545s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.582677770 (474h4m3.453760815s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.588464799 (474h4m3.459547844s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.712544699 (474h4m3.583627744s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.851482769 (474h4m3.722565814s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.882255556 (474h4m3.753338601s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918839348 (474h4m3.789922403s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918864156 (474h4m3.789947201s) [seg_pos={0 0}, upload=416129]: Segment reached optimal threshold of 13 pieces. Dec 25 09:33:03.919269627 (474h4m3.790352672s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 09:33:03.919280899 (474h4m3.790363944s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919532328 (474h4m3.790615373s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection Dec 25 09:33:03.919541034 (474h4m3.790624079s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919588836 (474h4m3.790671871s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection Dec 25 09:33:03.919609945 (474h4m3.79069299s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919639131 (474h4m3.790722176s) [seg_pos={0 0}, upload=416129]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:33:03.919766364 (474h4m3.790849409s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:33:03.919775921 (474h4m3.790858977s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:33:04.215163028 (474h4m4.086246073s) [upload=416129]: Done uploading segments... Dec 25 09:33:04.215194258 (474h4m4.086277303s) [upload=416129]: Upload finished. err: <nil> Duration: 5.781855335s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X......................X.........X...........X............] duration: 6.045630698s trace: https://jaeger.dev.storj.io/trace/795e298d20174ea1 FAIL (2025-12-25 09:42:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ]: Done starting upload Dec 25 09:33:01.262724874 (474h4m1.133807919s) [upload=416129]: Uploading segments... Dec 25 09:33:01.262992493 (474h4m1.134075538s) [upload=416129]: Got next segment. Inline: false Dec 25 09:33:01.263043731 (474h4m1.134126776s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:33:01.263050604 (474h4m1.134133649s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:33:01.511625191 (474h4m1.382708226s) [seg_pos={0 0}, upload=416129]: Begin upload segment... Dec 25 09:33:01.511677211 (474h4m1.382760246s) [seg_pos={0 0}, upload=416129]: Done begin upload segment. Dec 25 09:33:01.511706086 (474h4m1.382789131s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.520762050 (474h4m1.391845095s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.522962850 (474h4m1.394045895s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.528425993 (474h4m1.399509038s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.531387723 (474h4m1.402470768s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.532761669 (474h4m1.403844715s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535007145 (474h4m1.40609019s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535436131 (474h4m1.406519176s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.536867767 (474h4m1.407950812s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.537459183 (474h4m1.408542228s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.538526586 (474h4m1.409609631s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.540585998 (474h4m1.411669043s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542237873 (474h4m1.413320948s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542542713 (474h4m1.413625748s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.545729362 (474h4m1.416812407s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.546027639 (474h4m1.417110684s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:03.159768330 (474h4m3.030851375s) [upload=416129]: Next returned nil segment Dec 25 09:33:03.159785773 (474h4m3.030868808s) [upload=416129]: Waiting for error group managing segments... Dec 25 09:33:03.547727689 (474h4m3.418810734s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.548444835 (474h4m3.41952789s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.551962984 (474h4m3.423046039s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.560368159 (474h4m3.431451204s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.562870935 (474h4m3.43395398s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.563062640 (474h4m3.434145705s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.573975500 (474h4m3.445058545s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.582677770 (474h4m3.453760815s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.588464799 (474h4m3.459547844s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.712544699 (474h4m3.583627744s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.851482769 (474h4m3.722565814s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.882255556 (474h4m3.753338601s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918839348 (474h4m3.789922403s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918864156 (474h4m3.789947201s) [seg_pos={0 0}, upload=416129]: Segment reached optimal threshold of 13 pieces. Dec 25 09:33:03.919269627 (474h4m3.790352672s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 09:33:03.919280899 (474h4m3.790363944s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919532328 (474h4m3.790615373s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection Dec 25 09:33:03.919541034 (474h4m3.790624079s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919588836 (474h4m3.790671871s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection Dec 25 09:33:03.919609945 (474h4m3.79069299s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919639131 (474h4m3.790722176s) [seg_pos={0 0}, upload=416129]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:33:03.919766364 (474h4m3.790849409s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:33:03.919775921 (474h4m3.790858977s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:33:04.215163028 (474h4m4.086246073s) [upload=416129]: Done uploading segments... Dec 25 09:33:04.215194258 (474h4m4.086277303s) [upload=416129]: Upload finished. err: <nil> Duration: 6.04565811s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 3.600717106s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:41:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.600742374s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X......................X.........X...........X...........] duration: 6.583711933s trace: https://jaeger.dev.storj.io/trace/795e298d20174ea1 FAIL (2025-12-25 09:41:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ]: Done starting upload Dec 25 09:33:01.262724874 (474h4m1.133807919s) [upload=416129]: Uploading segments... Dec 25 09:33:01.262992493 (474h4m1.134075538s) [upload=416129]: Got next segment. Inline: false Dec 25 09:33:01.263043731 (474h4m1.134126776s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:33:01.263050604 (474h4m1.134133649s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:33:01.511625191 (474h4m1.382708226s) [seg_pos={0 0}, upload=416129]: Begin upload segment... Dec 25 09:33:01.511677211 (474h4m1.382760246s) [seg_pos={0 0}, upload=416129]: Done begin upload segment. Dec 25 09:33:01.511706086 (474h4m1.382789131s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.520762050 (474h4m1.391845095s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.522962850 (474h4m1.394045895s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.528425993 (474h4m1.399509038s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.531387723 (474h4m1.402470768s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.532761669 (474h4m1.403844715s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535007145 (474h4m1.40609019s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535436131 (474h4m1.406519176s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.536867767 (474h4m1.407950812s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.537459183 (474h4m1.408542228s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.538526586 (474h4m1.409609631s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.540585998 (474h4m1.411669043s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542237873 (474h4m1.413320948s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542542713 (474h4m1.413625748s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.545729362 (474h4m1.416812407s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.546027639 (474h4m1.417110684s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:03.159768330 (474h4m3.030851375s) [upload=416129]: Next returned nil segment Dec 25 09:33:03.159785773 (474h4m3.030868808s) [upload=416129]: Waiting for error group managing segments... Dec 25 09:33:03.547727689 (474h4m3.418810734s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.548444835 (474h4m3.41952789s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.551962984 (474h4m3.423046039s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.560368159 (474h4m3.431451204s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.562870935 (474h4m3.43395398s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.563062640 (474h4m3.434145705s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.573975500 (474h4m3.445058545s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.582677770 (474h4m3.453760815s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.588464799 (474h4m3.459547844s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.712544699 (474h4m3.583627744s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.851482769 (474h4m3.722565814s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.882255556 (474h4m3.753338601s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918839348 (474h4m3.789922403s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918864156 (474h4m3.789947201s) [seg_pos={0 0}, upload=416129]: Segment reached optimal threshold of 13 pieces. Dec 25 09:33:03.919269627 (474h4m3.790352672s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 09:33:03.919280899 (474h4m3.790363944s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919532328 (474h4m3.790615373s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection Dec 25 09:33:03.919541034 (474h4m3.790624079s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919588836 (474h4m3.790671871s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection Dec 25 09:33:03.919609945 (474h4m3.79069299s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919639131 (474h4m3.790722176s) [seg_pos={0 0}, upload=416129]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:33:03.919766364 (474h4m3.790849409s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:33:03.919775921 (474h4m3.790858977s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:33:04.215163028 (474h4m4.086246073s) [upload=416129]: Done uploading segments... Dec 25 09:33:04.215194258 (474h4m4.086277303s) [upload=416129]: Upload finished. err: <nil> Duration: 6.58374098s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.444839428s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:41:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.444866118s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.49102344s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:40:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.491049028s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X......................X.........X...........X..........] duration: 5.830613878s trace: https://jaeger.dev.storj.io/trace/795e298d20174ea1 FAIL (2025-12-25 09:40:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ]: Done starting upload Dec 25 09:33:01.262724874 (474h4m1.133807919s) [upload=416129]: Uploading segments... Dec 25 09:33:01.262992493 (474h4m1.134075538s) [upload=416129]: Got next segment. Inline: false Dec 25 09:33:01.263043731 (474h4m1.134126776s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:33:01.263050604 (474h4m1.134133649s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:33:01.511625191 (474h4m1.382708226s) [seg_pos={0 0}, upload=416129]: Begin upload segment... Dec 25 09:33:01.511677211 (474h4m1.382760246s) [seg_pos={0 0}, upload=416129]: Done begin upload segment. Dec 25 09:33:01.511706086 (474h4m1.382789131s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.520762050 (474h4m1.391845095s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.522962850 (474h4m1.394045895s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.528425993 (474h4m1.399509038s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.531387723 (474h4m1.402470768s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.532761669 (474h4m1.403844715s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535007145 (474h4m1.40609019s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535436131 (474h4m1.406519176s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.536867767 (474h4m1.407950812s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.537459183 (474h4m1.408542228s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.538526586 (474h4m1.409609631s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.540585998 (474h4m1.411669043s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542237873 (474h4m1.413320948s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542542713 (474h4m1.413625748s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.545729362 (474h4m1.416812407s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.546027639 (474h4m1.417110684s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:03.159768330 (474h4m3.030851375s) [upload=416129]: Next returned nil segment Dec 25 09:33:03.159785773 (474h4m3.030868808s) [upload=416129]: Waiting for error group managing segments... Dec 25 09:33:03.547727689 (474h4m3.418810734s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.548444835 (474h4m3.41952789s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.551962984 (474h4m3.423046039s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.560368159 (474h4m3.431451204s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.562870935 (474h4m3.43395398s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.563062640 (474h4m3.434145705s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.573975500 (474h4m3.445058545s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.582677770 (474h4m3.453760815s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.588464799 (474h4m3.459547844s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.712544699 (474h4m3.583627744s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.851482769 (474h4m3.722565814s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.882255556 (474h4m3.753338601s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918839348 (474h4m3.789922403s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918864156 (474h4m3.789947201s) [seg_pos={0 0}, upload=416129]: Segment reached optimal threshold of 13 pieces. Dec 25 09:33:03.919269627 (474h4m3.790352672s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 09:33:03.919280899 (474h4m3.790363944s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919532328 (474h4m3.790615373s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection Dec 25 09:33:03.919541034 (474h4m3.790624079s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919588836 (474h4m3.790671871s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection Dec 25 09:33:03.919609945 (474h4m3.79069299s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919639131 (474h4m3.790722176s) [seg_pos={0 0}, upload=416129]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:33:03.919766364 (474h4m3.790849409s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:33:03.919775921 (474h4m3.790858977s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:33:04.215163028 (474h4m4.086246073s) [upload=416129]: Done uploading segments... Dec 25 09:33:04.215194258 (474h4m4.086277303s) [upload=416129]: Upload finished. err: <nil> Duration: 5.830640129s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X......................X.........X...........X.........] duration: 6.280059008s trace: https://jaeger.dev.storj.io/trace/795e298d20174ea1 FAIL (2025-12-25 09:40:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ]: Done starting upload Dec 25 09:33:01.262724874 (474h4m1.133807919s) [upload=416129]: Uploading segments... Dec 25 09:33:01.262992493 (474h4m1.134075538s) [upload=416129]: Got next segment. Inline: false Dec 25 09:33:01.263043731 (474h4m1.134126776s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:33:01.263050604 (474h4m1.134133649s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:33:01.511625191 (474h4m1.382708226s) [seg_pos={0 0}, upload=416129]: Begin upload segment... Dec 25 09:33:01.511677211 (474h4m1.382760246s) [seg_pos={0 0}, upload=416129]: Done begin upload segment. Dec 25 09:33:01.511706086 (474h4m1.382789131s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.520762050 (474h4m1.391845095s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.522962850 (474h4m1.394045895s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.528425993 (474h4m1.399509038s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.531387723 (474h4m1.402470768s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.532761669 (474h4m1.403844715s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535007145 (474h4m1.40609019s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535436131 (474h4m1.406519176s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.536867767 (474h4m1.407950812s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.537459183 (474h4m1.408542228s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.538526586 (474h4m1.409609631s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.540585998 (474h4m1.411669043s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542237873 (474h4m1.413320948s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542542713 (474h4m1.413625748s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.545729362 (474h4m1.416812407s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.546027639 (474h4m1.417110684s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:03.159768330 (474h4m3.030851375s) [upload=416129]: Next returned nil segment Dec 25 09:33:03.159785773 (474h4m3.030868808s) [upload=416129]: Waiting for error group managing segments... Dec 25 09:33:03.547727689 (474h4m3.418810734s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.548444835 (474h4m3.41952789s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.551962984 (474h4m3.423046039s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.560368159 (474h4m3.431451204s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.562870935 (474h4m3.43395398s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.563062640 (474h4m3.434145705s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.573975500 (474h4m3.445058545s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.582677770 (474h4m3.453760815s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.588464799 (474h4m3.459547844s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.712544699 (474h4m3.583627744s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.851482769 (474h4m3.722565814s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.882255556 (474h4m3.753338601s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918839348 (474h4m3.789922403s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918864156 (474h4m3.789947201s) [seg_pos={0 0}, upload=416129]: Segment reached optimal threshold of 13 pieces. Dec 25 09:33:03.919269627 (474h4m3.790352672s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 09:33:03.919280899 (474h4m3.790363944s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919532328 (474h4m3.790615373s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection Dec 25 09:33:03.919541034 (474h4m3.790624079s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919588836 (474h4m3.790671871s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection Dec 25 09:33:03.919609945 (474h4m3.79069299s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919639131 (474h4m3.790722176s) [seg_pos={0 0}, upload=416129]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:33:03.919766364 (474h4m3.790849409s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:33:03.919775921 (474h4m3.790858977s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:33:04.215163028 (474h4m4.086246073s) [upload=416129]: Done uploading segments... Dec 25 09:33:04.215194258 (474h4m4.086277303s) [upload=416129]: Upload finished. err: <nil> Duration: 6.280090947s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.564908109s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:39:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.564932375s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X......................X.........X...........X........] duration: 5.707417843s trace: https://jaeger.dev.storj.io/trace/795e298d20174ea1 FAIL (2025-12-25 09:39:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ]: Done starting upload Dec 25 09:33:01.262724874 (474h4m1.133807919s) [upload=416129]: Uploading segments... Dec 25 09:33:01.262992493 (474h4m1.134075538s) [upload=416129]: Got next segment. Inline: false Dec 25 09:33:01.263043731 (474h4m1.134126776s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:33:01.263050604 (474h4m1.134133649s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:33:01.511625191 (474h4m1.382708226s) [seg_pos={0 0}, upload=416129]: Begin upload segment... Dec 25 09:33:01.511677211 (474h4m1.382760246s) [seg_pos={0 0}, upload=416129]: Done begin upload segment. Dec 25 09:33:01.511706086 (474h4m1.382789131s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.520762050 (474h4m1.391845095s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.522962850 (474h4m1.394045895s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.528425993 (474h4m1.399509038s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.531387723 (474h4m1.402470768s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.532761669 (474h4m1.403844715s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535007145 (474h4m1.40609019s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535436131 (474h4m1.406519176s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.536867767 (474h4m1.407950812s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.537459183 (474h4m1.408542228s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.538526586 (474h4m1.409609631s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.540585998 (474h4m1.411669043s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542237873 (474h4m1.413320948s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542542713 (474h4m1.413625748s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.545729362 (474h4m1.416812407s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.546027639 (474h4m1.417110684s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:03.159768330 (474h4m3.030851375s) [upload=416129]: Next returned nil segment Dec 25 09:33:03.159785773 (474h4m3.030868808s) [upload=416129]: Waiting for error group managing segments... Dec 25 09:33:03.547727689 (474h4m3.418810734s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.548444835 (474h4m3.41952789s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.551962984 (474h4m3.423046039s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.560368159 (474h4m3.431451204s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.562870935 (474h4m3.43395398s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.563062640 (474h4m3.434145705s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.573975500 (474h4m3.445058545s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.582677770 (474h4m3.453760815s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.588464799 (474h4m3.459547844s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.712544699 (474h4m3.583627744s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.851482769 (474h4m3.722565814s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.882255556 (474h4m3.753338601s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918839348 (474h4m3.789922403s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918864156 (474h4m3.789947201s) [seg_pos={0 0}, upload=416129]: Segment reached optimal threshold of 13 pieces. Dec 25 09:33:03.919269627 (474h4m3.790352672s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 09:33:03.919280899 (474h4m3.790363944s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919532328 (474h4m3.790615373s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection Dec 25 09:33:03.919541034 (474h4m3.790624079s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919588836 (474h4m3.790671871s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection Dec 25 09:33:03.919609945 (474h4m3.79069299s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919639131 (474h4m3.790722176s) [seg_pos={0 0}, upload=416129]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:33:03.919766364 (474h4m3.790849409s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:33:03.919775921 (474h4m3.790858977s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:33:04.215163028 (474h4m4.086246073s) [upload=416129]: Done uploading segments... Dec 25 09:33:04.215194258 (474h4m4.086277303s) [upload=416129]: Upload finished. err: <nil> Duration: 5.707445346s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 3.304217982s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:38:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.304242199s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X......................X.........X...........X.......] duration: 5.773469396s trace: https://jaeger.dev.storj.io/trace/795e298d20174ea1 FAIL (2025-12-25 09:38:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ]: Done starting upload Dec 25 09:33:01.262724874 (474h4m1.133807919s) [upload=416129]: Uploading segments... Dec 25 09:33:01.262992493 (474h4m1.134075538s) [upload=416129]: Got next segment. Inline: false Dec 25 09:33:01.263043731 (474h4m1.134126776s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:33:01.263050604 (474h4m1.134133649s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:33:01.511625191 (474h4m1.382708226s) [seg_pos={0 0}, upload=416129]: Begin upload segment... Dec 25 09:33:01.511677211 (474h4m1.382760246s) [seg_pos={0 0}, upload=416129]: Done begin upload segment. Dec 25 09:33:01.511706086 (474h4m1.382789131s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.520762050 (474h4m1.391845095s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.522962850 (474h4m1.394045895s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.528425993 (474h4m1.399509038s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.531387723 (474h4m1.402470768s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.532761669 (474h4m1.403844715s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535007145 (474h4m1.40609019s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535436131 (474h4m1.406519176s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.536867767 (474h4m1.407950812s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.537459183 (474h4m1.408542228s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.538526586 (474h4m1.409609631s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.540585998 (474h4m1.411669043s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542237873 (474h4m1.413320948s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542542713 (474h4m1.413625748s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.545729362 (474h4m1.416812407s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.546027639 (474h4m1.417110684s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:03.159768330 (474h4m3.030851375s) [upload=416129]: Next returned nil segment Dec 25 09:33:03.159785773 (474h4m3.030868808s) [upload=416129]: Waiting for error group managing segments... Dec 25 09:33:03.547727689 (474h4m3.418810734s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.548444835 (474h4m3.41952789s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.551962984 (474h4m3.423046039s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.560368159 (474h4m3.431451204s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.562870935 (474h4m3.43395398s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.563062640 (474h4m3.434145705s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.573975500 (474h4m3.445058545s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.582677770 (474h4m3.453760815s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.588464799 (474h4m3.459547844s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.712544699 (474h4m3.583627744s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.851482769 (474h4m3.722565814s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.882255556 (474h4m3.753338601s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918839348 (474h4m3.789922403s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918864156 (474h4m3.789947201s) [seg_pos={0 0}, upload=416129]: Segment reached optimal threshold of 13 pieces. Dec 25 09:33:03.919269627 (474h4m3.790352672s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 09:33:03.919280899 (474h4m3.790363944s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919532328 (474h4m3.790615373s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection Dec 25 09:33:03.919541034 (474h4m3.790624079s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919588836 (474h4m3.790671871s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection Dec 25 09:33:03.919609945 (474h4m3.79069299s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919639131 (474h4m3.790722176s) [seg_pos={0 0}, upload=416129]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:33:03.919766364 (474h4m3.790849409s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:33:03.919775921 (474h4m3.790858977s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:33:04.215163028 (474h4m4.086246073s) [upload=416129]: Done uploading segments... Dec 25 09:33:04.215194258 (474h4m4.086277303s) [upload=416129]: Upload finished. err: <nil> Duration: 5.773497612s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.411816576s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:37:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.411840162s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X......................X.........X...........X......] duration: 5.846659034s trace: https://jaeger.dev.storj.io/trace/795e298d20174ea1 FAIL (2025-12-25 09:37:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ]: Done starting upload Dec 25 09:33:01.262724874 (474h4m1.133807919s) [upload=416129]: Uploading segments... Dec 25 09:33:01.262992493 (474h4m1.134075538s) [upload=416129]: Got next segment. Inline: false Dec 25 09:33:01.263043731 (474h4m1.134126776s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:33:01.263050604 (474h4m1.134133649s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:33:01.511625191 (474h4m1.382708226s) [seg_pos={0 0}, upload=416129]: Begin upload segment... Dec 25 09:33:01.511677211 (474h4m1.382760246s) [seg_pos={0 0}, upload=416129]: Done begin upload segment. Dec 25 09:33:01.511706086 (474h4m1.382789131s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.520762050 (474h4m1.391845095s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.522962850 (474h4m1.394045895s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.528425993 (474h4m1.399509038s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.531387723 (474h4m1.402470768s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.532761669 (474h4m1.403844715s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535007145 (474h4m1.40609019s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535436131 (474h4m1.406519176s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.536867767 (474h4m1.407950812s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.537459183 (474h4m1.408542228s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.538526586 (474h4m1.409609631s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.540585998 (474h4m1.411669043s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542237873 (474h4m1.413320948s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542542713 (474h4m1.413625748s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.545729362 (474h4m1.416812407s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.546027639 (474h4m1.417110684s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:03.159768330 (474h4m3.030851375s) [upload=416129]: Next returned nil segment Dec 25 09:33:03.159785773 (474h4m3.030868808s) [upload=416129]: Waiting for error group managing segments... Dec 25 09:33:03.547727689 (474h4m3.418810734s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.548444835 (474h4m3.41952789s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.551962984 (474h4m3.423046039s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.560368159 (474h4m3.431451204s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.562870935 (474h4m3.43395398s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.563062640 (474h4m3.434145705s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.573975500 (474h4m3.445058545s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.582677770 (474h4m3.453760815s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.588464799 (474h4m3.459547844s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.712544699 (474h4m3.583627744s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.851482769 (474h4m3.722565814s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.882255556 (474h4m3.753338601s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918839348 (474h4m3.789922403s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918864156 (474h4m3.789947201s) [seg_pos={0 0}, upload=416129]: Segment reached optimal threshold of 13 pieces. Dec 25 09:33:03.919269627 (474h4m3.790352672s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 09:33:03.919280899 (474h4m3.790363944s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919532328 (474h4m3.790615373s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection Dec 25 09:33:03.919541034 (474h4m3.790624079s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919588836 (474h4m3.790671871s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection Dec 25 09:33:03.919609945 (474h4m3.79069299s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919639131 (474h4m3.790722176s) [seg_pos={0 0}, upload=416129]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:33:03.919766364 (474h4m3.790849409s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:33:03.919775921 (474h4m3.790858977s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:33:04.215163028 (474h4m4.086246073s) [upload=416129]: Done uploading segments... Dec 25 09:33:04.215194258 (474h4m4.086277303s) [upload=416129]: Upload finished. err: <nil> Duration: 5.846749816s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X......................X.........X...........X.....] duration: 5.773941837s trace: https://jaeger.dev.storj.io/trace/795e298d20174ea1 FAIL (2025-12-25 09:36:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ]: Done starting upload Dec 25 09:33:01.262724874 (474h4m1.133807919s) [upload=416129]: Uploading segments... Dec 25 09:33:01.262992493 (474h4m1.134075538s) [upload=416129]: Got next segment. Inline: false Dec 25 09:33:01.263043731 (474h4m1.134126776s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:33:01.263050604 (474h4m1.134133649s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:33:01.511625191 (474h4m1.382708226s) [seg_pos={0 0}, upload=416129]: Begin upload segment... Dec 25 09:33:01.511677211 (474h4m1.382760246s) [seg_pos={0 0}, upload=416129]: Done begin upload segment. Dec 25 09:33:01.511706086 (474h4m1.382789131s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.520762050 (474h4m1.391845095s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.522962850 (474h4m1.394045895s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.528425993 (474h4m1.399509038s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.531387723 (474h4m1.402470768s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.532761669 (474h4m1.403844715s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535007145 (474h4m1.40609019s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535436131 (474h4m1.406519176s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.536867767 (474h4m1.407950812s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.537459183 (474h4m1.408542228s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.538526586 (474h4m1.409609631s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.540585998 (474h4m1.411669043s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542237873 (474h4m1.413320948s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542542713 (474h4m1.413625748s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.545729362 (474h4m1.416812407s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.546027639 (474h4m1.417110684s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:03.159768330 (474h4m3.030851375s) [upload=416129]: Next returned nil segment Dec 25 09:33:03.159785773 (474h4m3.030868808s) [upload=416129]: Waiting for error group managing segments... Dec 25 09:33:03.547727689 (474h4m3.418810734s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.548444835 (474h4m3.41952789s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.551962984 (474h4m3.423046039s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.560368159 (474h4m3.431451204s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.562870935 (474h4m3.43395398s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.563062640 (474h4m3.434145705s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.573975500 (474h4m3.445058545s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.582677770 (474h4m3.453760815s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.588464799 (474h4m3.459547844s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.712544699 (474h4m3.583627744s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.851482769 (474h4m3.722565814s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.882255556 (474h4m3.753338601s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918839348 (474h4m3.789922403s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918864156 (474h4m3.789947201s) [seg_pos={0 0}, upload=416129]: Segment reached optimal threshold of 13 pieces. Dec 25 09:33:03.919269627 (474h4m3.790352672s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 09:33:03.919280899 (474h4m3.790363944s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919532328 (474h4m3.790615373s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection Dec 25 09:33:03.919541034 (474h4m3.790624079s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919588836 (474h4m3.790671871s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection Dec 25 09:33:03.919609945 (474h4m3.79069299s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919639131 (474h4m3.790722176s) [seg_pos={0 0}, upload=416129]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:33:03.919766364 (474h4m3.790849409s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:33:03.919775921 (474h4m3.790858977s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:33:04.215163028 (474h4m4.086246073s) [upload=416129]: Done uploading segments... Dec 25 09:33:04.215194258 (474h4m4.086277303s) [upload=416129]: Upload finished. err: <nil> Duration: 5.773970282s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.755842279s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:36:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.755871604s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.58505917s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:36:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.585086752s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X......................X.........X...........X....] duration: 6.017967813s trace: https://jaeger.dev.storj.io/trace/795e298d20174ea1 FAIL (2025-12-25 09:35:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ]: Done starting upload Dec 25 09:33:01.262724874 (474h4m1.133807919s) [upload=416129]: Uploading segments... Dec 25 09:33:01.262992493 (474h4m1.134075538s) [upload=416129]: Got next segment. Inline: false Dec 25 09:33:01.263043731 (474h4m1.134126776s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:33:01.263050604 (474h4m1.134133649s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:33:01.511625191 (474h4m1.382708226s) [seg_pos={0 0}, upload=416129]: Begin upload segment... Dec 25 09:33:01.511677211 (474h4m1.382760246s) [seg_pos={0 0}, upload=416129]: Done begin upload segment. Dec 25 09:33:01.511706086 (474h4m1.382789131s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.520762050 (474h4m1.391845095s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.522962850 (474h4m1.394045895s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.528425993 (474h4m1.399509038s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.531387723 (474h4m1.402470768s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.532761669 (474h4m1.403844715s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535007145 (474h4m1.40609019s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535436131 (474h4m1.406519176s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.536867767 (474h4m1.407950812s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.537459183 (474h4m1.408542228s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.538526586 (474h4m1.409609631s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.540585998 (474h4m1.411669043s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542237873 (474h4m1.413320948s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542542713 (474h4m1.413625748s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.545729362 (474h4m1.416812407s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.546027639 (474h4m1.417110684s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:03.159768330 (474h4m3.030851375s) [upload=416129]: Next returned nil segment Dec 25 09:33:03.159785773 (474h4m3.030868808s) [upload=416129]: Waiting for error group managing segments... Dec 25 09:33:03.547727689 (474h4m3.418810734s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.548444835 (474h4m3.41952789s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.551962984 (474h4m3.423046039s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.560368159 (474h4m3.431451204s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.562870935 (474h4m3.43395398s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.563062640 (474h4m3.434145705s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.573975500 (474h4m3.445058545s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.582677770 (474h4m3.453760815s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.588464799 (474h4m3.459547844s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.712544699 (474h4m3.583627744s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.851482769 (474h4m3.722565814s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.882255556 (474h4m3.753338601s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918839348 (474h4m3.789922403s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918864156 (474h4m3.789947201s) [seg_pos={0 0}, upload=416129]: Segment reached optimal threshold of 13 pieces. Dec 25 09:33:03.919269627 (474h4m3.790352672s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 09:33:03.919280899 (474h4m3.790363944s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919532328 (474h4m3.790615373s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection Dec 25 09:33:03.919541034 (474h4m3.790624079s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919588836 (474h4m3.790671871s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection Dec 25 09:33:03.919609945 (474h4m3.79069299s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919639131 (474h4m3.790722176s) [seg_pos={0 0}, upload=416129]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:33:03.919766364 (474h4m3.790849409s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:33:03.919775921 (474h4m3.790858977s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:33:04.215163028 (474h4m4.086246073s) [upload=416129]: Done uploading segments... Dec 25 09:33:04.215194258 (474h4m4.086277303s) [upload=416129]: Upload finished. err: <nil> Duration: 6.017994342s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X......................X.........X...........X...] duration: 5.742346074s trace: https://jaeger.dev.storj.io/trace/795e298d20174ea1 FAIL (2025-12-25 09:35:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ]: Done starting upload Dec 25 09:33:01.262724874 (474h4m1.133807919s) [upload=416129]: Uploading segments... Dec 25 09:33:01.262992493 (474h4m1.134075538s) [upload=416129]: Got next segment. Inline: false Dec 25 09:33:01.263043731 (474h4m1.134126776s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:33:01.263050604 (474h4m1.134133649s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:33:01.511625191 (474h4m1.382708226s) [seg_pos={0 0}, upload=416129]: Begin upload segment... Dec 25 09:33:01.511677211 (474h4m1.382760246s) [seg_pos={0 0}, upload=416129]: Done begin upload segment. Dec 25 09:33:01.511706086 (474h4m1.382789131s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.520762050 (474h4m1.391845095s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.522962850 (474h4m1.394045895s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.528425993 (474h4m1.399509038s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.531387723 (474h4m1.402470768s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.532761669 (474h4m1.403844715s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535007145 (474h4m1.40609019s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535436131 (474h4m1.406519176s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.536867767 (474h4m1.407950812s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.537459183 (474h4m1.408542228s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.538526586 (474h4m1.409609631s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.540585998 (474h4m1.411669043s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542237873 (474h4m1.413320948s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542542713 (474h4m1.413625748s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.545729362 (474h4m1.416812407s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.546027639 (474h4m1.417110684s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:03.159768330 (474h4m3.030851375s) [upload=416129]: Next returned nil segment Dec 25 09:33:03.159785773 (474h4m3.030868808s) [upload=416129]: Waiting for error group managing segments... Dec 25 09:33:03.547727689 (474h4m3.418810734s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.548444835 (474h4m3.41952789s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.551962984 (474h4m3.423046039s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.560368159 (474h4m3.431451204s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.562870935 (474h4m3.43395398s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.563062640 (474h4m3.434145705s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.573975500 (474h4m3.445058545s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.582677770 (474h4m3.453760815s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.588464799 (474h4m3.459547844s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.712544699 (474h4m3.583627744s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.851482769 (474h4m3.722565814s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.882255556 (474h4m3.753338601s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918839348 (474h4m3.789922403s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918864156 (474h4m3.789947201s) [seg_pos={0 0}, upload=416129]: Segment reached optimal threshold of 13 pieces. Dec 25 09:33:03.919269627 (474h4m3.790352672s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 09:33:03.919280899 (474h4m3.790363944s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919532328 (474h4m3.790615373s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection Dec 25 09:33:03.919541034 (474h4m3.790624079s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919588836 (474h4m3.790671871s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection Dec 25 09:33:03.919609945 (474h4m3.79069299s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919639131 (474h4m3.790722176s) [seg_pos={0 0}, upload=416129]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:33:03.919766364 (474h4m3.790849409s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:33:03.919775921 (474h4m3.790858977s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:33:04.215163028 (474h4m4.086246073s) [upload=416129]: Done uploading segments... Dec 25 09:33:04.215194258 (474h4m4.086277303s) [upload=416129]: Upload finished. err: <nil> Duration: 5.742373637s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.429908475s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:35:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.430005852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X......................X.........X...........X..] duration: 5.913330991s trace: https://jaeger.dev.storj.io/trace/795e298d20174ea1 FAIL (2025-12-25 09:34:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ]: Done starting upload Dec 25 09:33:01.262724874 (474h4m1.133807919s) [upload=416129]: Uploading segments... Dec 25 09:33:01.262992493 (474h4m1.134075538s) [upload=416129]: Got next segment. Inline: false Dec 25 09:33:01.263043731 (474h4m1.134126776s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:33:01.263050604 (474h4m1.134133649s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:33:01.511625191 (474h4m1.382708226s) [seg_pos={0 0}, upload=416129]: Begin upload segment... Dec 25 09:33:01.511677211 (474h4m1.382760246s) [seg_pos={0 0}, upload=416129]: Done begin upload segment. Dec 25 09:33:01.511706086 (474h4m1.382789131s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.520762050 (474h4m1.391845095s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.522962850 (474h4m1.394045895s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.528425993 (474h4m1.399509038s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.531387723 (474h4m1.402470768s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.532761669 (474h4m1.403844715s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535007145 (474h4m1.40609019s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535436131 (474h4m1.406519176s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.536867767 (474h4m1.407950812s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.537459183 (474h4m1.408542228s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.538526586 (474h4m1.409609631s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.540585998 (474h4m1.411669043s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542237873 (474h4m1.413320948s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542542713 (474h4m1.413625748s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.545729362 (474h4m1.416812407s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.546027639 (474h4m1.417110684s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:03.159768330 (474h4m3.030851375s) [upload=416129]: Next returned nil segment Dec 25 09:33:03.159785773 (474h4m3.030868808s) [upload=416129]: Waiting for error group managing segments... Dec 25 09:33:03.547727689 (474h4m3.418810734s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.548444835 (474h4m3.41952789s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.551962984 (474h4m3.423046039s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.560368159 (474h4m3.431451204s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.562870935 (474h4m3.43395398s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.563062640 (474h4m3.434145705s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.573975500 (474h4m3.445058545s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.582677770 (474h4m3.453760815s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.588464799 (474h4m3.459547844s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.712544699 (474h4m3.583627744s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.851482769 (474h4m3.722565814s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.882255556 (474h4m3.753338601s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918839348 (474h4m3.789922403s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918864156 (474h4m3.789947201s) [seg_pos={0 0}, upload=416129]: Segment reached optimal threshold of 13 pieces. Dec 25 09:33:03.919269627 (474h4m3.790352672s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 09:33:03.919280899 (474h4m3.790363944s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919532328 (474h4m3.790615373s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection Dec 25 09:33:03.919541034 (474h4m3.790624079s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919588836 (474h4m3.790671871s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection Dec 25 09:33:03.919609945 (474h4m3.79069299s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919639131 (474h4m3.790722176s) [seg_pos={0 0}, upload=416129]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:33:03.919766364 (474h4m3.790849409s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:33:03.919775921 (474h4m3.790858977s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:33:04.215163028 (474h4m4.086246073s) [upload=416129]: Done uploading segments... Dec 25 09:33:04.215194258 (474h4m4.086277303s) [upload=416129]: Upload finished. err: <nil> Duration: 5.913416253s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.493981644s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:34:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.494012201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X......................X.........X...........X.] duration: 5.723911951s trace: https://jaeger.dev.storj.io/trace/795e298d20174ea1 FAIL (2025-12-25 09:33:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ]: Done starting upload Dec 25 09:33:01.262724874 (474h4m1.133807919s) [upload=416129]: Uploading segments... Dec 25 09:33:01.262992493 (474h4m1.134075538s) [upload=416129]: Got next segment. Inline: false Dec 25 09:33:01.263043731 (474h4m1.134126776s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:33:01.263050604 (474h4m1.134133649s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:33:01.511625191 (474h4m1.382708226s) [seg_pos={0 0}, upload=416129]: Begin upload segment... Dec 25 09:33:01.511677211 (474h4m1.382760246s) [seg_pos={0 0}, upload=416129]: Done begin upload segment. Dec 25 09:33:01.511706086 (474h4m1.382789131s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.520762050 (474h4m1.391845095s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.522962850 (474h4m1.394045895s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.528425993 (474h4m1.399509038s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.531387723 (474h4m1.402470768s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.532761669 (474h4m1.403844715s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535007145 (474h4m1.40609019s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535436131 (474h4m1.406519176s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.536867767 (474h4m1.407950812s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.537459183 (474h4m1.408542228s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.538526586 (474h4m1.409609631s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.540585998 (474h4m1.411669043s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542237873 (474h4m1.413320948s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542542713 (474h4m1.413625748s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.545729362 (474h4m1.416812407s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.546027639 (474h4m1.417110684s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:03.159768330 (474h4m3.030851375s) [upload=416129]: Next returned nil segment Dec 25 09:33:03.159785773 (474h4m3.030868808s) [upload=416129]: Waiting for error group managing segments... Dec 25 09:33:03.547727689 (474h4m3.418810734s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.548444835 (474h4m3.41952789s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.551962984 (474h4m3.423046039s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.560368159 (474h4m3.431451204s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.562870935 (474h4m3.43395398s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.563062640 (474h4m3.434145705s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.573975500 (474h4m3.445058545s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.582677770 (474h4m3.453760815s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.588464799 (474h4m3.459547844s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.712544699 (474h4m3.583627744s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.851482769 (474h4m3.722565814s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.882255556 (474h4m3.753338601s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918839348 (474h4m3.789922403s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918864156 (474h4m3.789947201s) [seg_pos={0 0}, upload=416129]: Segment reached optimal threshold of 13 pieces. Dec 25 09:33:03.919269627 (474h4m3.790352672s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 09:33:03.919280899 (474h4m3.790363944s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919532328 (474h4m3.790615373s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection Dec 25 09:33:03.919541034 (474h4m3.790624079s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919588836 (474h4m3.790671871s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection Dec 25 09:33:03.919609945 (474h4m3.79069299s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919639131 (474h4m3.790722176s) [seg_pos={0 0}, upload=416129]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:33:03.919766364 (474h4m3.790849409s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:33:03.919775921 (474h4m3.790858977s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:33:04.215163028 (474h4m4.086246073s) [upload=416129]: Done uploading segments... Dec 25 09:33:04.215194258 (474h4m4.086277303s) [upload=416129]: Upload finished. err: <nil> Duration: 5.723947528s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.............X......................X.........X...........X] duration: 30.003262245s trace: https://jaeger.dev.storj.io/trace/795e298d20174ea1 FAIL (2025-12-25 09:33:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ]: Done starting upload Dec 25 09:33:01.262724874 (474h4m1.133807919s) [upload=416129]: Uploading segments... Dec 25 09:33:01.262992493 (474h4m1.134075538s) [upload=416129]: Got next segment. Inline: false Dec 25 09:33:01.263043731 (474h4m1.134126776s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:33:01.263050604 (474h4m1.134133649s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:33:01.511625191 (474h4m1.382708226s) [seg_pos={0 0}, upload=416129]: Begin upload segment... Dec 25 09:33:01.511677211 (474h4m1.382760246s) [seg_pos={0 0}, upload=416129]: Done begin upload segment. Dec 25 09:33:01.511706086 (474h4m1.382789131s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.520762050 (474h4m1.391845095s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.522962850 (474h4m1.394045895s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.528425993 (474h4m1.399509038s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.531387723 (474h4m1.402470768s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.532761669 (474h4m1.403844715s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535007145 (474h4m1.40609019s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535436131 (474h4m1.406519176s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.536867767 (474h4m1.407950812s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.537459183 (474h4m1.408542228s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.538526586 (474h4m1.409609631s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.540585998 (474h4m1.411669043s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542237873 (474h4m1.413320948s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542542713 (474h4m1.413625748s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.545729362 (474h4m1.416812407s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.546027639 (474h4m1.417110684s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:03.159768330 (474h4m3.030851375s) [upload=416129]: Next returned nil segment Dec 25 09:33:03.159785773 (474h4m3.030868808s) [upload=416129]: Waiting for error group managing segments... Dec 25 09:33:03.547727689 (474h4m3.418810734s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.548444835 (474h4m3.41952789s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.551962984 (474h4m3.423046039s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.560368159 (474h4m3.431451204s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.562870935 (474h4m3.43395398s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.563062640 (474h4m3.434145705s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.573975500 (474h4m3.445058545s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.582677770 (474h4m3.453760815s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.588464799 (474h4m3.459547844s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.712544699 (474h4m3.583627744s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.851482769 (474h4m3.722565814s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.882255556 (474h4m3.753338601s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918839348 (474h4m3.789922403s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918864156 (474h4m3.789947201s) [seg_pos={0 0}, upload=416129]: Segment reached optimal threshold of 13 pieces. Dec 25 09:33:03.919269627 (474h4m3.790352672s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 09:33:03.919280899 (474h4m3.790363944s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919532328 (474h4m3.790615373s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection Dec 25 09:33:03.919541034 (474h4m3.790624079s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919588836 (474h4m3.790671871s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection Dec 25 09:33:03.919609945 (474h4m3.79069299s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919639131 (474h4m3.790722176s) [seg_pos={0 0}, upload=416129]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:33:03.919766364 (474h4m3.790849409s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:33:03.919775921 (474h4m3.790858977s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:33:04.215163028 (474h4m4.086246073s) [upload=416129]: Done uploading segments... Dec 25 09:33:04.215194258 (474h4m4.086277303s) [upload=416129]: Upload finished. err: <nil> Duration: 30.003285481s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.580580261s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:33:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.580605639s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.466041552s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:32:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.466067943s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.............X......................X.........X...........] duration: 5.894580664s trace: https://jaeger.dev.storj.io/trace/4915bc3f760bc705 FAIL (2025-12-25 09:32:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ) context deadline exceeded ======LOGS====== Dec 25 09:23:50.224552770 (473h54m50.095635805s) [upload=415994]: Starting upload Dec 25 09:23:50.224607274 (473h54m50.095690309s) [upload=415994]: Done starting upload Dec 25 09:23:50.224644124 (473h54m50.095727159s) [upload=415994]: Uploading segments... Dec 25 09:23:50.224806143 (473h54m50.095889178s) [upload=415994]: Got next segment. Inline: false Dec 25 09:23:50.224833665 (473h54m50.09591671s) [upload=415994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:23:50.224837824 (473h54m50.095920859s) [upload=415994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:23:50.470907829 (473h54m50.341990874s) [seg_pos={0 0}, upload=415994]: Begin upload segment... Dec 25 09:23:50.470957613 (473h54m50.342040658s) [seg_pos={0 0}, upload=415994]: Done begin upload segment. Dec 25 09:23:50.470997900 (473h54m50.342080935s) [noise=true , address=145.79.227.29:4942 , piece_id=UGG5ZUP62B6N7VB5O2TDLJ3CYVWVUQEHZOY362KXPV7NDGFOTYUA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471033848 (473h54m50.342116883s) [noise=true , address=145.79.227.33:4871 , piece_id=5WE37TDD26EFHRPFPTZG4KUEXQCOEFUKIGQJUXXT5TBFWAJHSQLQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471502841 (473h54m50.342585877s) [noise=true , address=145.79.227.39:4013 , piece_id=D3XGGDEJDYKRNPYM3SKASVQPEYC5LJ5UXSPJQMW5KII2RLYBP2QA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471852267 (473h54m50.342935302s) [noise=true , address=145.79.227.37:4565 , piece_id=CTCRPDXKNRPZLI35HE4O2N7JHJYGGSYXNPSTIVQ23XBXBF43VMEA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472267968 (473h54m50.343351004s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472596213 (473h54m50.343679248s) [noise=true , address=145.79.227.41:4086 , piece_id=CVHACN2KJAUKOOGR7MTCMYT725L4DN4LCIUWNGK5JBXJAMNB72WA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472902737 (473h54m50.343985772s) [noise=true , address=145.79.227.8:4330 , piece_id=ZPJLYRARNKIJMMJ5EKCJXCI3CROCTJFT6ERHZ2UZTXQMW6SYWBYA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.473202198 (473h54m50.344285233s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.473549970 (473h54m50.344633005s) [noise=true , address=145.79.227.50:4374 , piece_id=FGY3UU2VI342L2GMBJBKD3I4TJTUY65AQ2AJENRM2DSH2MTWHVGA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.475125379 (473h54m50.346208414s) [noise=true , address=145.79.227.43:4159 , piece_id=CONWY4VFJO44KLFRJJHFKYBJIENXR5OZPWCANWFR23CPHAO5XHMA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.476176541 (473h54m50.347259586s) [noise=true , address=145.79.227.15:4875 , piece_id=SFECL2JQUYDHMJJNBXMHOZML25G6HF4246UDPSOGXBJ4BXEIC4ZA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.477297886 (473h54m50.348380921s) [noise=true , address=45.91.116.9:4261 , piece_id=CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.478326435 (473h54m50.34940947s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.479334314 (473h54m50.350417349s) [noise=true , address=145.79.227.51:4410 , piece_id=XXQRFI64ML5YCVX6GIEILCMINWD77IK2ORGOKV67RMSRYYG5F3SA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.480341863 (473h54m50.351424898s) [noise=true , address=45.91.116.6:4974 , piece_id=VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.482939428 (473h54m50.354022463s) [noise=true , address=145.79.227.45:4229 , piece_id=CTVIS33TC3CWQADPBXH2QTE4YHHNPSDQSUEFROENDMVY7XO7C3TA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:52.139736479 (473h54m52.010819514s) [upload=415994]: Next returned nil segment Dec 25 09:23:52.139763281 (473h54m52.010846316s) [upload=415994]: Waiting for error group managing segments... Dec 25 09:23:52.476853834 (473h54m52.347936869s) [noise=true , address=145.79.227.37:4565 , piece_id=CTCRPDXKNRPZLI35HE4O2N7JHJYGGSYXNPSTIVQ23XBXBF43VMEA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.478814327 (473h54m52.349897372s) [noise=true , address=145.79.227.41:4086 , piece_id=CVHACN2KJAUKOOGR7MTCMYT725L4DN4LCIUWNGK5JBXJAMNB72WA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.483839445 (473h54m52.3549225s) [noise=true , address=145.79.227.50:4374 , piece_id=FGY3UU2VI342L2GMBJBKD3I4TJTUY65AQ2AJENRM2DSH2MTWHVGA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.487109452 (473h54m52.358192497s) [noise=true , address=145.79.227.8:4330 , piece_id=ZPJLYRARNKIJMMJ5EKCJXCI3CROCTJFT6ERHZ2UZTXQMW6SYWBYA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.491064052 (473h54m52.362147097s) [noise=true , address=145.79.227.15:4875 , piece_id=SFECL2JQUYDHMJJNBXMHOZML25G6HF4246UDPSOGXBJ4BXEIC4ZA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.494245760 (473h54m52.365328795s) [noise=true , address=145.79.227.45:4229 , piece_id=CTVIS33TC3CWQADPBXH2QTE4YHHNPSDQSUEFROENDMVY7XO7C3TA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.494657655 (473h54m52.36574069s) [noise=true , address=145.79.227.43:4159 , piece_id=CONWY4VFJO44KLFRJJHFKYBJIENXR5OZPWCANWFR23CPHAO5XHMA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.499742347 (473h54m52.370825382s) [noise=true , address=145.79.227.39:4013 , piece_id=D3XGGDEJDYKRNPYM3SKASVQPEYC5LJ5UXSPJQMW5KII2RLYBP2QA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.502818174 (473h54m52.373901249s) [noise=true , address=145.79.227.29:4942 , piece_id=UGG5ZUP62B6N7VB5O2TDLJ3CYVWVUQEHZOY362KXPV7NDGFOTYUA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.520302817 (473h54m52.391385852s) [noise=true , address=145.79.227.51:4410 , piece_id=XXQRFI64ML5YCVX6GIEILCMINWD77IK2ORGOKV67RMSRYYG5F3SA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.752925748 (473h54m52.624008783s) [noise=true , address=45.91.116.9:4261 , piece_id=CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.882738941 (473h54m52.753821976s) [noise=true , address=145.79.227.33:4871 , piece_id=5WE37TDD26EFHRPFPTZG4KUEXQCOEFUKIGQJUXXT5TBFWAJHSQLQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.990394505 (473h54m52.86147755s) [noise=true , address=45.91.116.6:4974 , piece_id=VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.990414122 (473h54m52.861497167s) [seg_pos={0 0}, upload=415994]: Segment reached optimal threshold of 13 pieces. Dec 25 09:23:52.990564689 (473h54m52.861647724s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection Dec 25 09:23:52.990569888 (473h54m52.861652924s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990691491 (473h54m52.861774526s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 25 09:23:52.990696901 (473h54m52.861779946s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990859900 (473h54m52.861942935s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 25 09:23:52.990865131 (473h54m52.861948166s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990880981 (473h54m52.861964016s) [seg_pos={0 0}, upload=415994]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:23:52.990954581 (473h54m52.862037626s) [upload=415994]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:23:52.990961794 (473h54m52.86204483s) [upload=415994]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:23:53.296758561 (473h54m53.167841606s) [upload=415994]: Done uploading segments... Dec 25 09:23:53.296784670 (473h54m53.167867705s) [upload=415994]: Upload finished. err: <nil> Duration: 5.894610792s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.............X......................X.........X..........] duration: 5.776793194s trace: https://jaeger.dev.storj.io/trace/4915bc3f760bc705 FAIL (2025-12-25 09:31:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ) context deadline exceeded ======LOGS====== Dec 25 09:23:50.224552770 (473h54m50.095635805s) [upload=415994]: Starting upload Dec 25 09:23:50.224607274 (473h54m50.095690309s) [upload=415994]: Done starting upload Dec 25 09:23:50.224644124 (473h54m50.095727159s) [upload=415994]: Uploading segments... Dec 25 09:23:50.224806143 (473h54m50.095889178s) [upload=415994]: Got next segment. Inline: false Dec 25 09:23:50.224833665 (473h54m50.09591671s) [upload=415994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:23:50.224837824 (473h54m50.095920859s) [upload=415994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:23:50.470907829 (473h54m50.341990874s) [seg_pos={0 0}, upload=415994]: Begin upload segment... Dec 25 09:23:50.470957613 (473h54m50.342040658s) [seg_pos={0 0}, upload=415994]: Done begin upload segment. Dec 25 09:23:50.470997900 (473h54m50.342080935s) [noise=true , address=145.79.227.29:4942 , piece_id=UGG5ZUP62B6N7VB5O2TDLJ3CYVWVUQEHZOY362KXPV7NDGFOTYUA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471033848 (473h54m50.342116883s) [noise=true , address=145.79.227.33:4871 , piece_id=5WE37TDD26EFHRPFPTZG4KUEXQCOEFUKIGQJUXXT5TBFWAJHSQLQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471502841 (473h54m50.342585877s) [noise=true , address=145.79.227.39:4013 , piece_id=D3XGGDEJDYKRNPYM3SKASVQPEYC5LJ5UXSPJQMW5KII2RLYBP2QA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471852267 (473h54m50.342935302s) [noise=true , address=145.79.227.37:4565 , piece_id=CTCRPDXKNRPZLI35HE4O2N7JHJYGGSYXNPSTIVQ23XBXBF43VMEA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472267968 (473h54m50.343351004s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472596213 (473h54m50.343679248s) [noise=true , address=145.79.227.41:4086 , piece_id=CVHACN2KJAUKOOGR7MTCMYT725L4DN4LCIUWNGK5JBXJAMNB72WA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472902737 (473h54m50.343985772s) [noise=true , address=145.79.227.8:4330 , piece_id=ZPJLYRARNKIJMMJ5EKCJXCI3CROCTJFT6ERHZ2UZTXQMW6SYWBYA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.473202198 (473h54m50.344285233s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.473549970 (473h54m50.344633005s) [noise=true , address=145.79.227.50:4374 , piece_id=FGY3UU2VI342L2GMBJBKD3I4TJTUY65AQ2AJENRM2DSH2MTWHVGA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.475125379 (473h54m50.346208414s) [noise=true , address=145.79.227.43:4159 , piece_id=CONWY4VFJO44KLFRJJHFKYBJIENXR5OZPWCANWFR23CPHAO5XHMA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.476176541 (473h54m50.347259586s) [noise=true , address=145.79.227.15:4875 , piece_id=SFECL2JQUYDHMJJNBXMHOZML25G6HF4246UDPSOGXBJ4BXEIC4ZA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.477297886 (473h54m50.348380921s) [noise=true , address=45.91.116.9:4261 , piece_id=CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.478326435 (473h54m50.34940947s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.479334314 (473h54m50.350417349s) [noise=true , address=145.79.227.51:4410 , piece_id=XXQRFI64ML5YCVX6GIEILCMINWD77IK2ORGOKV67RMSRYYG5F3SA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.480341863 (473h54m50.351424898s) [noise=true , address=45.91.116.6:4974 , piece_id=VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.482939428 (473h54m50.354022463s) [noise=true , address=145.79.227.45:4229 , piece_id=CTVIS33TC3CWQADPBXH2QTE4YHHNPSDQSUEFROENDMVY7XO7C3TA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:52.139736479 (473h54m52.010819514s) [upload=415994]: Next returned nil segment Dec 25 09:23:52.139763281 (473h54m52.010846316s) [upload=415994]: Waiting for error group managing segments... Dec 25 09:23:52.476853834 (473h54m52.347936869s) [noise=true , address=145.79.227.37:4565 , piece_id=CTCRPDXKNRPZLI35HE4O2N7JHJYGGSYXNPSTIVQ23XBXBF43VMEA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.478814327 (473h54m52.349897372s) [noise=true , address=145.79.227.41:4086 , piece_id=CVHACN2KJAUKOOGR7MTCMYT725L4DN4LCIUWNGK5JBXJAMNB72WA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.483839445 (473h54m52.3549225s) [noise=true , address=145.79.227.50:4374 , piece_id=FGY3UU2VI342L2GMBJBKD3I4TJTUY65AQ2AJENRM2DSH2MTWHVGA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.487109452 (473h54m52.358192497s) [noise=true , address=145.79.227.8:4330 , piece_id=ZPJLYRARNKIJMMJ5EKCJXCI3CROCTJFT6ERHZ2UZTXQMW6SYWBYA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.491064052 (473h54m52.362147097s) [noise=true , address=145.79.227.15:4875 , piece_id=SFECL2JQUYDHMJJNBXMHOZML25G6HF4246UDPSOGXBJ4BXEIC4ZA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.494245760 (473h54m52.365328795s) [noise=true , address=145.79.227.45:4229 , piece_id=CTVIS33TC3CWQADPBXH2QTE4YHHNPSDQSUEFROENDMVY7XO7C3TA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.494657655 (473h54m52.36574069s) [noise=true , address=145.79.227.43:4159 , piece_id=CONWY4VFJO44KLFRJJHFKYBJIENXR5OZPWCANWFR23CPHAO5XHMA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.499742347 (473h54m52.370825382s) [noise=true , address=145.79.227.39:4013 , piece_id=D3XGGDEJDYKRNPYM3SKASVQPEYC5LJ5UXSPJQMW5KII2RLYBP2QA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.502818174 (473h54m52.373901249s) [noise=true , address=145.79.227.29:4942 , piece_id=UGG5ZUP62B6N7VB5O2TDLJ3CYVWVUQEHZOY362KXPV7NDGFOTYUA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.520302817 (473h54m52.391385852s) [noise=true , address=145.79.227.51:4410 , piece_id=XXQRFI64ML5YCVX6GIEILCMINWD77IK2ORGOKV67RMSRYYG5F3SA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.752925748 (473h54m52.624008783s) [noise=true , address=45.91.116.9:4261 , piece_id=CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.882738941 (473h54m52.753821976s) [noise=true , address=145.79.227.33:4871 , piece_id=5WE37TDD26EFHRPFPTZG4KUEXQCOEFUKIGQJUXXT5TBFWAJHSQLQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.990394505 (473h54m52.86147755s) [noise=true , address=45.91.116.6:4974 , piece_id=VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.990414122 (473h54m52.861497167s) [seg_pos={0 0}, upload=415994]: Segment reached optimal threshold of 13 pieces. Dec 25 09:23:52.990564689 (473h54m52.861647724s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection Dec 25 09:23:52.990569888 (473h54m52.861652924s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990691491 (473h54m52.861774526s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 25 09:23:52.990696901 (473h54m52.861779946s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990859900 (473h54m52.861942935s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 25 09:23:52.990865131 (473h54m52.861948166s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990880981 (473h54m52.861964016s) [seg_pos={0 0}, upload=415994]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:23:52.990954581 (473h54m52.862037626s) [upload=415994]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:23:52.990961794 (473h54m52.86204483s) [upload=415994]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:23:53.296758561 (473h54m53.167841606s) [upload=415994]: Done uploading segments... Dec 25 09:23:53.296784670 (473h54m53.167867705s) [upload=415994]: Upload finished. err: <nil> Duration: 5.776813923s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.50289267s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:31:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.502921124s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.............X......................X.........X.........] duration: 5.632369378s trace: https://jaeger.dev.storj.io/trace/4915bc3f760bc705 FAIL (2025-12-25 09:31:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ) context deadline exceeded ======LOGS====== Dec 25 09:23:50.224552770 (473h54m50.095635805s) [upload=415994]: Starting upload Dec 25 09:23:50.224607274 (473h54m50.095690309s) [upload=415994]: Done starting upload Dec 25 09:23:50.224644124 (473h54m50.095727159s) [upload=415994]: Uploading segments... Dec 25 09:23:50.224806143 (473h54m50.095889178s) [upload=415994]: Got next segment. Inline: false Dec 25 09:23:50.224833665 (473h54m50.09591671s) [upload=415994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:23:50.224837824 (473h54m50.095920859s) [upload=415994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:23:50.470907829 (473h54m50.341990874s) [seg_pos={0 0}, upload=415994]: Begin upload segment... Dec 25 09:23:50.470957613 (473h54m50.342040658s) [seg_pos={0 0}, upload=415994]: Done begin upload segment. Dec 25 09:23:50.470997900 (473h54m50.342080935s) [noise=true , address=145.79.227.29:4942 , piece_id=UGG5ZUP62B6N7VB5O2TDLJ3CYVWVUQEHZOY362KXPV7NDGFOTYUA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471033848 (473h54m50.342116883s) [noise=true , address=145.79.227.33:4871 , piece_id=5WE37TDD26EFHRPFPTZG4KUEXQCOEFUKIGQJUXXT5TBFWAJHSQLQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471502841 (473h54m50.342585877s) [noise=true , address=145.79.227.39:4013 , piece_id=D3XGGDEJDYKRNPYM3SKASVQPEYC5LJ5UXSPJQMW5KII2RLYBP2QA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471852267 (473h54m50.342935302s) [noise=true , address=145.79.227.37:4565 , piece_id=CTCRPDXKNRPZLI35HE4O2N7JHJYGGSYXNPSTIVQ23XBXBF43VMEA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472267968 (473h54m50.343351004s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472596213 (473h54m50.343679248s) [noise=true , address=145.79.227.41:4086 , piece_id=CVHACN2KJAUKOOGR7MTCMYT725L4DN4LCIUWNGK5JBXJAMNB72WA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472902737 (473h54m50.343985772s) [noise=true , address=145.79.227.8:4330 , piece_id=ZPJLYRARNKIJMMJ5EKCJXCI3CROCTJFT6ERHZ2UZTXQMW6SYWBYA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.473202198 (473h54m50.344285233s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.473549970 (473h54m50.344633005s) [noise=true , address=145.79.227.50:4374 , piece_id=FGY3UU2VI342L2GMBJBKD3I4TJTUY65AQ2AJENRM2DSH2MTWHVGA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.475125379 (473h54m50.346208414s) [noise=true , address=145.79.227.43:4159 , piece_id=CONWY4VFJO44KLFRJJHFKYBJIENXR5OZPWCANWFR23CPHAO5XHMA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.476176541 (473h54m50.347259586s) [noise=true , address=145.79.227.15:4875 , piece_id=SFECL2JQUYDHMJJNBXMHOZML25G6HF4246UDPSOGXBJ4BXEIC4ZA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.477297886 (473h54m50.348380921s) [noise=true , address=45.91.116.9:4261 , piece_id=CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.478326435 (473h54m50.34940947s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.479334314 (473h54m50.350417349s) [noise=true , address=145.79.227.51:4410 , piece_id=XXQRFI64ML5YCVX6GIEILCMINWD77IK2ORGOKV67RMSRYYG5F3SA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.480341863 (473h54m50.351424898s) [noise=true , address=45.91.116.6:4974 , piece_id=VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.482939428 (473h54m50.354022463s) [noise=true , address=145.79.227.45:4229 , piece_id=CTVIS33TC3CWQADPBXH2QTE4YHHNPSDQSUEFROENDMVY7XO7C3TA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:52.139736479 (473h54m52.010819514s) [upload=415994]: Next returned nil segment Dec 25 09:23:52.139763281 (473h54m52.010846316s) [upload=415994]: Waiting for error group managing segments... Dec 25 09:23:52.476853834 (473h54m52.347936869s) [noise=true , address=145.79.227.37:4565 , piece_id=CTCRPDXKNRPZLI35HE4O2N7JHJYGGSYXNPSTIVQ23XBXBF43VMEA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.478814327 (473h54m52.349897372s) [noise=true , address=145.79.227.41:4086 , piece_id=CVHACN2KJAUKOOGR7MTCMYT725L4DN4LCIUWNGK5JBXJAMNB72WA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.483839445 (473h54m52.3549225s) [noise=true , address=145.79.227.50:4374 , piece_id=FGY3UU2VI342L2GMBJBKD3I4TJTUY65AQ2AJENRM2DSH2MTWHVGA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.487109452 (473h54m52.358192497s) [noise=true , address=145.79.227.8:4330 , piece_id=ZPJLYRARNKIJMMJ5EKCJXCI3CROCTJFT6ERHZ2UZTXQMW6SYWBYA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.491064052 (473h54m52.362147097s) [noise=true , address=145.79.227.15:4875 , piece_id=SFECL2JQUYDHMJJNBXMHOZML25G6HF4246UDPSOGXBJ4BXEIC4ZA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.494245760 (473h54m52.365328795s) [noise=true , address=145.79.227.45:4229 , piece_id=CTVIS33TC3CWQADPBXH2QTE4YHHNPSDQSUEFROENDMVY7XO7C3TA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.494657655 (473h54m52.36574069s) [noise=true , address=145.79.227.43:4159 , piece_id=CONWY4VFJO44KLFRJJHFKYBJIENXR5OZPWCANWFR23CPHAO5XHMA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.499742347 (473h54m52.370825382s) [noise=true , address=145.79.227.39:4013 , piece_id=D3XGGDEJDYKRNPYM3SKASVQPEYC5LJ5UXSPJQMW5KII2RLYBP2QA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.502818174 (473h54m52.373901249s) [noise=true , address=145.79.227.29:4942 , piece_id=UGG5ZUP62B6N7VB5O2TDLJ3CYVWVUQEHZOY362KXPV7NDGFOTYUA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.520302817 (473h54m52.391385852s) [noise=true , address=145.79.227.51:4410 , piece_id=XXQRFI64ML5YCVX6GIEILCMINWD77IK2ORGOKV67RMSRYYG5F3SA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.752925748 (473h54m52.624008783s) [noise=true , address=45.91.116.9:4261 , piece_id=CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.882738941 (473h54m52.753821976s) [noise=true , address=145.79.227.33:4871 , piece_id=5WE37TDD26EFHRPFPTZG4KUEXQCOEFUKIGQJUXXT5TBFWAJHSQLQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.990394505 (473h54m52.86147755s) [noise=true , address=45.91.116.6:4974 , piece_id=VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.990414122 (473h54m52.861497167s) [seg_pos={0 0}, upload=415994]: Segment reached optimal threshold of 13 pieces. Dec 25 09:23:52.990564689 (473h54m52.861647724s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection Dec 25 09:23:52.990569888 (473h54m52.861652924s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990691491 (473h54m52.861774526s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 25 09:23:52.990696901 (473h54m52.861779946s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990859900 (473h54m52.861942935s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 25 09:23:52.990865131 (473h54m52.861948166s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990880981 (473h54m52.861964016s) [seg_pos={0 0}, upload=415994]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:23:52.990954581 (473h54m52.862037626s) [upload=415994]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:23:52.990961794 (473h54m52.86204483s) [upload=415994]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:23:53.296758561 (473h54m53.167841606s) [upload=415994]: Done uploading segments... Dec 25 09:23:53.296784670 (473h54m53.167867705s) [upload=415994]: Upload finished. err: <nil> Duration: 5.632405318s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.61162659s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:30:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.611653602s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.502223924s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:30:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.502255474s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.............X......................X.........X........] duration: 5.646714763s trace: https://jaeger.dev.storj.io/trace/4915bc3f760bc705 FAIL (2025-12-25 09:30:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ) context deadline exceeded ======LOGS====== Dec 25 09:23:50.224552770 (473h54m50.095635805s) [upload=415994]: Starting upload Dec 25 09:23:50.224607274 (473h54m50.095690309s) [upload=415994]: Done starting upload Dec 25 09:23:50.224644124 (473h54m50.095727159s) [upload=415994]: Uploading segments... Dec 25 09:23:50.224806143 (473h54m50.095889178s) [upload=415994]: Got next segment. Inline: false Dec 25 09:23:50.224833665 (473h54m50.09591671s) [upload=415994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:23:50.224837824 (473h54m50.095920859s) [upload=415994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:23:50.470907829 (473h54m50.341990874s) [seg_pos={0 0}, upload=415994]: Begin upload segment... Dec 25 09:23:50.470957613 (473h54m50.342040658s) [seg_pos={0 0}, upload=415994]: Done begin upload segment. Dec 25 09:23:50.470997900 (473h54m50.342080935s) [noise=true , address=145.79.227.29:4942 , piece_id=UGG5ZUP62B6N7VB5O2TDLJ3CYVWVUQEHZOY362KXPV7NDGFOTYUA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471033848 (473h54m50.342116883s) [noise=true , address=145.79.227.33:4871 , piece_id=5WE37TDD26EFHRPFPTZG4KUEXQCOEFUKIGQJUXXT5TBFWAJHSQLQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471502841 (473h54m50.342585877s) [noise=true , address=145.79.227.39:4013 , piece_id=D3XGGDEJDYKRNPYM3SKASVQPEYC5LJ5UXSPJQMW5KII2RLYBP2QA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471852267 (473h54m50.342935302s) [noise=true , address=145.79.227.37:4565 , piece_id=CTCRPDXKNRPZLI35HE4O2N7JHJYGGSYXNPSTIVQ23XBXBF43VMEA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472267968 (473h54m50.343351004s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472596213 (473h54m50.343679248s) [noise=true , address=145.79.227.41:4086 , piece_id=CVHACN2KJAUKOOGR7MTCMYT725L4DN4LCIUWNGK5JBXJAMNB72WA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472902737 (473h54m50.343985772s) [noise=true , address=145.79.227.8:4330 , piece_id=ZPJLYRARNKIJMMJ5EKCJXCI3CROCTJFT6ERHZ2UZTXQMW6SYWBYA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.473202198 (473h54m50.344285233s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.473549970 (473h54m50.344633005s) [noise=true , address=145.79.227.50:4374 , piece_id=FGY3UU2VI342L2GMBJBKD3I4TJTUY65AQ2AJENRM2DSH2MTWHVGA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.475125379 (473h54m50.346208414s) [noise=true , address=145.79.227.43:4159 , piece_id=CONWY4VFJO44KLFRJJHFKYBJIENXR5OZPWCANWFR23CPHAO5XHMA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.476176541 (473h54m50.347259586s) [noise=true , address=145.79.227.15:4875 , piece_id=SFECL2JQUYDHMJJNBXMHOZML25G6HF4246UDPSOGXBJ4BXEIC4ZA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.477297886 (473h54m50.348380921s) [noise=true , address=45.91.116.9:4261 , piece_id=CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.478326435 (473h54m50.34940947s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.479334314 (473h54m50.350417349s) [noise=true , address=145.79.227.51:4410 , piece_id=XXQRFI64ML5YCVX6GIEILCMINWD77IK2ORGOKV67RMSRYYG5F3SA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.480341863 (473h54m50.351424898s) [noise=true , address=45.91.116.6:4974 , piece_id=VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.482939428 (473h54m50.354022463s) [noise=true , address=145.79.227.45:4229 , piece_id=CTVIS33TC3CWQADPBXH2QTE4YHHNPSDQSUEFROENDMVY7XO7C3TA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:52.139736479 (473h54m52.010819514s) [upload=415994]: Next returned nil segment Dec 25 09:23:52.139763281 (473h54m52.010846316s) [upload=415994]: Waiting for error group managing segments... Dec 25 09:23:52.476853834 (473h54m52.347936869s) [noise=true , address=145.79.227.37:4565 , piece_id=CTCRPDXKNRPZLI35HE4O2N7JHJYGGSYXNPSTIVQ23XBXBF43VMEA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.478814327 (473h54m52.349897372s) [noise=true , address=145.79.227.41:4086 , piece_id=CVHACN2KJAUKOOGR7MTCMYT725L4DN4LCIUWNGK5JBXJAMNB72WA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.483839445 (473h54m52.3549225s) [noise=true , address=145.79.227.50:4374 , piece_id=FGY3UU2VI342L2GMBJBKD3I4TJTUY65AQ2AJENRM2DSH2MTWHVGA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.487109452 (473h54m52.358192497s) [noise=true , address=145.79.227.8:4330 , piece_id=ZPJLYRARNKIJMMJ5EKCJXCI3CROCTJFT6ERHZ2UZTXQMW6SYWBYA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.491064052 (473h54m52.362147097s) [noise=true , address=145.79.227.15:4875 , piece_id=SFECL2JQUYDHMJJNBXMHOZML25G6HF4246UDPSOGXBJ4BXEIC4ZA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.494245760 (473h54m52.365328795s) [noise=true , address=145.79.227.45:4229 , piece_id=CTVIS33TC3CWQADPBXH2QTE4YHHNPSDQSUEFROENDMVY7XO7C3TA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.494657655 (473h54m52.36574069s) [noise=true , address=145.79.227.43:4159 , piece_id=CONWY4VFJO44KLFRJJHFKYBJIENXR5OZPWCANWFR23CPHAO5XHMA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.499742347 (473h54m52.370825382s) [noise=true , address=145.79.227.39:4013 , piece_id=D3XGGDEJDYKRNPYM3SKASVQPEYC5LJ5UXSPJQMW5KII2RLYBP2QA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.502818174 (473h54m52.373901249s) [noise=true , address=145.79.227.29:4942 , piece_id=UGG5ZUP62B6N7VB5O2TDLJ3CYVWVUQEHZOY362KXPV7NDGFOTYUA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.520302817 (473h54m52.391385852s) [noise=true , address=145.79.227.51:4410 , piece_id=XXQRFI64ML5YCVX6GIEILCMINWD77IK2ORGOKV67RMSRYYG5F3SA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.752925748 (473h54m52.624008783s) [noise=true , address=45.91.116.9:4261 , piece_id=CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.882738941 (473h54m52.753821976s) [noise=true , address=145.79.227.33:4871 , piece_id=5WE37TDD26EFHRPFPTZG4KUEXQCOEFUKIGQJUXXT5TBFWAJHSQLQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.990394505 (473h54m52.86147755s) [noise=true , address=45.91.116.6:4974 , piece_id=VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.990414122 (473h54m52.861497167s) [seg_pos={0 0}, upload=415994]: Segment reached optimal threshold of 13 pieces. Dec 25 09:23:52.990564689 (473h54m52.861647724s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection Dec 25 09:23:52.990569888 (473h54m52.861652924s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990691491 (473h54m52.861774526s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 25 09:23:52.990696901 (473h54m52.861779946s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990859900 (473h54m52.861942935s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 25 09:23:52.990865131 (473h54m52.861948166s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990880981 (473h54m52.861964016s) [seg_pos={0 0}, upload=415994]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:23:52.990954581 (473h54m52.862037626s) [upload=415994]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:23:52.990961794 (473h54m52.86204483s) [upload=415994]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:23:53.296758561 (473h54m53.167841606s) [upload=415994]: Done uploading segments... Dec 25 09:23:53.296784670 (473h54m53.167867705s) [upload=415994]: Upload finished. err: <nil> Duration: 5.646740581s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.............X......................X.........X.......] duration: 5.746042516s trace: https://jaeger.dev.storj.io/trace/4915bc3f760bc705 FAIL (2025-12-25 09:29:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ) context deadline exceeded ======LOGS====== Dec 25 09:23:50.224552770 (473h54m50.095635805s) [upload=415994]: Starting upload Dec 25 09:23:50.224607274 (473h54m50.095690309s) [upload=415994]: Done starting upload Dec 25 09:23:50.224644124 (473h54m50.095727159s) [upload=415994]: Uploading segments... Dec 25 09:23:50.224806143 (473h54m50.095889178s) [upload=415994]: Got next segment. Inline: false Dec 25 09:23:50.224833665 (473h54m50.09591671s) [upload=415994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:23:50.224837824 (473h54m50.095920859s) [upload=415994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:23:50.470907829 (473h54m50.341990874s) [seg_pos={0 0}, upload=415994]: Begin upload segment... Dec 25 09:23:50.470957613 (473h54m50.342040658s) [seg_pos={0 0}, upload=415994]: Done begin upload segment. Dec 25 09:23:50.470997900 (473h54m50.342080935s) [noise=true , address=145.79.227.29:4942 , piece_id=UGG5ZUP62B6N7VB5O2TDLJ3CYVWVUQEHZOY362KXPV7NDGFOTYUA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471033848 (473h54m50.342116883s) [noise=true , address=145.79.227.33:4871 , piece_id=5WE37TDD26EFHRPFPTZG4KUEXQCOEFUKIGQJUXXT5TBFWAJHSQLQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471502841 (473h54m50.342585877s) [noise=true , address=145.79.227.39:4013 , piece_id=D3XGGDEJDYKRNPYM3SKASVQPEYC5LJ5UXSPJQMW5KII2RLYBP2QA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471852267 (473h54m50.342935302s) [noise=true , address=145.79.227.37:4565 , piece_id=CTCRPDXKNRPZLI35HE4O2N7JHJYGGSYXNPSTIVQ23XBXBF43VMEA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472267968 (473h54m50.343351004s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472596213 (473h54m50.343679248s) [noise=true , address=145.79.227.41:4086 , piece_id=CVHACN2KJAUKOOGR7MTCMYT725L4DN4LCIUWNGK5JBXJAMNB72WA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472902737 (473h54m50.343985772s) [noise=true , address=145.79.227.8:4330 , piece_id=ZPJLYRARNKIJMMJ5EKCJXCI3CROCTJFT6ERHZ2UZTXQMW6SYWBYA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.473202198 (473h54m50.344285233s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.473549970 (473h54m50.344633005s) [noise=true , address=145.79.227.50:4374 , piece_id=FGY3UU2VI342L2GMBJBKD3I4TJTUY65AQ2AJENRM2DSH2MTWHVGA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.475125379 (473h54m50.346208414s) [noise=true , address=145.79.227.43:4159 , piece_id=CONWY4VFJO44KLFRJJHFKYBJIENXR5OZPWCANWFR23CPHAO5XHMA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.476176541 (473h54m50.347259586s) [noise=true , address=145.79.227.15:4875 , piece_id=SFECL2JQUYDHMJJNBXMHOZML25G6HF4246UDPSOGXBJ4BXEIC4ZA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.477297886 (473h54m50.348380921s) [noise=true , address=45.91.116.9:4261 , piece_id=CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.478326435 (473h54m50.34940947s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.479334314 (473h54m50.350417349s) [noise=true , address=145.79.227.51:4410 , piece_id=XXQRFI64ML5YCVX6GIEILCMINWD77IK2ORGOKV67RMSRYYG5F3SA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.480341863 (473h54m50.351424898s) [noise=true , address=45.91.116.6:4974 , piece_id=VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.482939428 (473h54m50.354022463s) [noise=true , address=145.79.227.45:4229 , piece_id=CTVIS33TC3CWQADPBXH2QTE4YHHNPSDQSUEFROENDMVY7XO7C3TA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:52.139736479 (473h54m52.010819514s) [upload=415994]: Next returned nil segment Dec 25 09:23:52.139763281 (473h54m52.010846316s) [upload=415994]: Waiting for error group managing segments... Dec 25 09:23:52.476853834 (473h54m52.347936869s) [noise=true , address=145.79.227.37:4565 , piece_id=CTCRPDXKNRPZLI35HE4O2N7JHJYGGSYXNPSTIVQ23XBXBF43VMEA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.478814327 (473h54m52.349897372s) [noise=true , address=145.79.227.41:4086 , piece_id=CVHACN2KJAUKOOGR7MTCMYT725L4DN4LCIUWNGK5JBXJAMNB72WA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.483839445 (473h54m52.3549225s) [noise=true , address=145.79.227.50:4374 , piece_id=FGY3UU2VI342L2GMBJBKD3I4TJTUY65AQ2AJENRM2DSH2MTWHVGA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.487109452 (473h54m52.358192497s) [noise=true , address=145.79.227.8:4330 , piece_id=ZPJLYRARNKIJMMJ5EKCJXCI3CROCTJFT6ERHZ2UZTXQMW6SYWBYA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.491064052 (473h54m52.362147097s) [noise=true , address=145.79.227.15:4875 , piece_id=SFECL2JQUYDHMJJNBXMHOZML25G6HF4246UDPSOGXBJ4BXEIC4ZA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.494245760 (473h54m52.365328795s) [noise=true , address=145.79.227.45:4229 , piece_id=CTVIS33TC3CWQADPBXH2QTE4YHHNPSDQSUEFROENDMVY7XO7C3TA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.494657655 (473h54m52.36574069s) [noise=true , address=145.79.227.43:4159 , piece_id=CONWY4VFJO44KLFRJJHFKYBJIENXR5OZPWCANWFR23CPHAO5XHMA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.499742347 (473h54m52.370825382s) [noise=true , address=145.79.227.39:4013 , piece_id=D3XGGDEJDYKRNPYM3SKASVQPEYC5LJ5UXSPJQMW5KII2RLYBP2QA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.502818174 (473h54m52.373901249s) [noise=true , address=145.79.227.29:4942 , piece_id=UGG5ZUP62B6N7VB5O2TDLJ3CYVWVUQEHZOY362KXPV7NDGFOTYUA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.520302817 (473h54m52.391385852s) [noise=true , address=145.79.227.51:4410 , piece_id=XXQRFI64ML5YCVX6GIEILCMINWD77IK2ORGOKV67RMSRYYG5F3SA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.752925748 (473h54m52.624008783s) [noise=true , address=45.91.116.9:4261 , piece_id=CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.882738941 (473h54m52.753821976s) [noise=true , address=145.79.227.33:4871 , piece_id=5WE37TDD26EFHRPFPTZG4KUEXQCOEFUKIGQJUXXT5TBFWAJHSQLQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.990394505 (473h54m52.86147755s) [noise=true , address=45.91.116.6:4974 , piece_id=VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.990414122 (473h54m52.861497167s) [seg_pos={0 0}, upload=415994]: Segment reached optimal threshold of 13 pieces. Dec 25 09:23:52.990564689 (473h54m52.861647724s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection Dec 25 09:23:52.990569888 (473h54m52.861652924s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990691491 (473h54m52.861774526s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 25 09:23:52.990696901 (473h54m52.861779946s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990859900 (473h54m52.861942935s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 25 09:23:52.990865131 (473h54m52.861948166s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990880981 (473h54m52.861964016s) [seg_pos={0 0}, upload=415994]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:23:52.990954581 (473h54m52.862037626s) [upload=415994]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:23:52.990961794 (473h54m52.86204483s) [upload=415994]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:23:53.296758561 (473h54m53.167841606s) [upload=415994]: Done uploading segments... Dec 25 09:23:53.296784670 (473h54m53.167867705s) [upload=415994]: Upload finished. err: <nil> Duration: 5.746065779s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.470821919s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:29:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.470845042s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.919255506s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:28:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.919277669s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.567406319s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:28:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.567435086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.....X.............X......................X.........X......] duration: 5.659037392s trace: https://jaeger.dev.storj.io/trace/4915bc3f760bc705 FAIL (2025-12-25 09:28:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ) context deadline exceeded ======LOGS====== Dec 25 09:23:50.224552770 (473h54m50.095635805s) [upload=415994]: Starting upload Dec 25 09:23:50.224607274 (473h54m50.095690309s) [upload=415994]: Done starting upload Dec 25 09:23:50.224644124 (473h54m50.095727159s) [upload=415994]: Uploading segments... Dec 25 09:23:50.224806143 (473h54m50.095889178s) [upload=415994]: Got next segment. Inline: false Dec 25 09:23:50.224833665 (473h54m50.09591671s) [upload=415994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:23:50.224837824 (473h54m50.095920859s) [upload=415994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:23:50.470907829 (473h54m50.341990874s) [seg_pos={0 0}, upload=415994]: Begin upload segment... Dec 25 09:23:50.470957613 (473h54m50.342040658s) [seg_pos={0 0}, upload=415994]: Done begin upload segment. Dec 25 09:23:50.470997900 (473h54m50.342080935s) [noise=true , address=145.79.227.29:4942 , piece_id=UGG5ZUP62B6N7VB5O2TDLJ3CYVWVUQEHZOY362KXPV7NDGFOTYUA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471033848 (473h54m50.342116883s) [noise=true , address=145.79.227.33:4871 , piece_id=5WE37TDD26EFHRPFPTZG4KUEXQCOEFUKIGQJUXXT5TBFWAJHSQLQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471502841 (473h54m50.342585877s) [noise=true , address=145.79.227.39:4013 , piece_id=D3XGGDEJDYKRNPYM3SKASVQPEYC5LJ5UXSPJQMW5KII2RLYBP2QA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471852267 (473h54m50.342935302s) [noise=true , address=145.79.227.37:4565 , piece_id=CTCRPDXKNRPZLI35HE4O2N7JHJYGGSYXNPSTIVQ23XBXBF43VMEA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472267968 (473h54m50.343351004s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472596213 (473h54m50.343679248s) [noise=true , address=145.79.227.41:4086 , piece_id=CVHACN2KJAUKOOGR7MTCMYT725L4DN4LCIUWNGK5JBXJAMNB72WA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472902737 (473h54m50.343985772s) [noise=true , address=145.79.227.8:4330 , piece_id=ZPJLYRARNKIJMMJ5EKCJXCI3CROCTJFT6ERHZ2UZTXQMW6SYWBYA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.473202198 (473h54m50.344285233s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.473549970 (473h54m50.344633005s) [noise=true , address=145.79.227.50:4374 , piece_id=FGY3UU2VI342L2GMBJBKD3I4TJTUY65AQ2AJENRM2DSH2MTWHVGA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.475125379 (473h54m50.346208414s) [noise=true , address=145.79.227.43:4159 , piece_id=CONWY4VFJO44KLFRJJHFKYBJIENXR5OZPWCANWFR23CPHAO5XHMA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.476176541 (473h54m50.347259586s) [noise=true , address=145.79.227.15:4875 , piece_id=SFECL2JQUYDHMJJNBXMHOZML25G6HF4246UDPSOGXBJ4BXEIC4ZA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.477297886 (473h54m50.348380921s) [noise=true , address=45.91.116.9:4261 , piece_id=CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.478326435 (473h54m50.34940947s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.479334314 (473h54m50.350417349s) [noise=true , address=145.79.227.51:4410 , piece_id=XXQRFI64ML5YCVX6GIEILCMINWD77IK2ORGOKV67RMSRYYG5F3SA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.480341863 (473h54m50.351424898s) [noise=true , address=45.91.116.6:4974 , piece_id=VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.482939428 (473h54m50.354022463s) [noise=true , address=145.79.227.45:4229 , piece_id=CTVIS33TC3CWQADPBXH2QTE4YHHNPSDQSUEFROENDMVY7XO7C3TA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:52.139736479 (473h54m52.010819514s) [upload=415994]: Next returned nil segment Dec 25 09:23:52.139763281 (473h54m52.010846316s) [upload=415994]: Waiting for error group managing segments... Dec 25 09:23:52.476853834 (473h54m52.347936869s) [noise=true , address=145.79.227.37:4565 , piece_id=CTCRPDXKNRPZLI35HE4O2N7JHJYGGSYXNPSTIVQ23XBXBF43VMEA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.478814327 (473h54m52.349897372s) [noise=true , address=145.79.227.41:4086 , piece_id=CVHACN2KJAUKOOGR7MTCMYT725L4DN4LCIUWNGK5JBXJAMNB72WA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.483839445 (473h54m52.3549225s) [noise=true , address=145.79.227.50:4374 , piece_id=FGY3UU2VI342L2GMBJBKD3I4TJTUY65AQ2AJENRM2DSH2MTWHVGA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.487109452 (473h54m52.358192497s) [noise=true , address=145.79.227.8:4330 , piece_id=ZPJLYRARNKIJMMJ5EKCJXCI3CROCTJFT6ERHZ2UZTXQMW6SYWBYA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.491064052 (473h54m52.362147097s) [noise=true , address=145.79.227.15:4875 , piece_id=SFECL2JQUYDHMJJNBXMHOZML25G6HF4246UDPSOGXBJ4BXEIC4ZA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.494245760 (473h54m52.365328795s) [noise=true , address=145.79.227.45:4229 , piece_id=CTVIS33TC3CWQADPBXH2QTE4YHHNPSDQSUEFROENDMVY7XO7C3TA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.494657655 (473h54m52.36574069s) [noise=true , address=145.79.227.43:4159 , piece_id=CONWY4VFJO44KLFRJJHFKYBJIENXR5OZPWCANWFR23CPHAO5XHMA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.499742347 (473h54m52.370825382s) [noise=true , address=145.79.227.39:4013 , piece_id=D3XGGDEJDYKRNPYM3SKASVQPEYC5LJ5UXSPJQMW5KII2RLYBP2QA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.502818174 (473h54m52.373901249s) [noise=true , address=145.79.227.29:4942 , piece_id=UGG5ZUP62B6N7VB5O2TDLJ3CYVWVUQEHZOY362KXPV7NDGFOTYUA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.520302817 (473h54m52.391385852s) [noise=true , address=145.79.227.51:4410 , piece_id=XXQRFI64ML5YCVX6GIEILCMINWD77IK2ORGOKV67RMSRYYG5F3SA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.752925748 (473h54m52.624008783s) [noise=true , address=45.91.116.9:4261 , piece_id=CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.882738941 (473h54m52.753821976s) [noise=true , address=145.79.227.33:4871 , piece_id=5WE37TDD26EFHRPFPTZG4KUEXQCOEFUKIGQJUXXT5TBFWAJHSQLQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.990394505 (473h54m52.86147755s) [noise=true , address=45.91.116.6:4974 , piece_id=VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.990414122 (473h54m52.861497167s) [seg_pos={0 0}, upload=415994]: Segment reached optimal threshold of 13 pieces. Dec 25 09:23:52.990564689 (473h54m52.861647724s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection Dec 25 09:23:52.990569888 (473h54m52.861652924s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990691491 (473h54m52.861774526s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 25 09:23:52.990696901 (473h54m52.861779946s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990859900 (473h54m52.861942935s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 25 09:23:52.990865131 (473h54m52.861948166s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990880981 (473h54m52.861964016s) [seg_pos={0 0}, upload=415994]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:23:52.990954581 (473h54m52.862037626s) [upload=415994]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:23:52.990961794 (473h54m52.86204483s) [upload=415994]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:23:53.296758561 (473h54m53.167841606s) [upload=415994]: Done uploading segments... Dec 25 09:23:53.296784670 (473h54m53.167867705s) [upload=415994]: Upload finished. err: <nil> Duration: 5.659104559s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 4.448520269s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:28:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.448550126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.....X.............X......................X.........X.....] duration: 5.908956097s trace: https://jaeger.dev.storj.io/trace/4915bc3f760bc705 FAIL (2025-12-25 09:28:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ) context deadline exceeded ======LOGS====== Dec 25 09:23:50.224552770 (473h54m50.095635805s) [upload=415994]: Starting upload Dec 25 09:23:50.224607274 (473h54m50.095690309s) [upload=415994]: Done starting upload Dec 25 09:23:50.224644124 (473h54m50.095727159s) [upload=415994]: Uploading segments... Dec 25 09:23:50.224806143 (473h54m50.095889178s) [upload=415994]: Got next segment. Inline: false Dec 25 09:23:50.224833665 (473h54m50.09591671s) [upload=415994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:23:50.224837824 (473h54m50.095920859s) [upload=415994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:23:50.470907829 (473h54m50.341990874s) [seg_pos={0 0}, upload=415994]: Begin upload segment... Dec 25 09:23:50.470957613 (473h54m50.342040658s) [seg_pos={0 0}, upload=415994]: Done begin upload segment. Dec 25 09:23:50.470997900 (473h54m50.342080935s) [noise=true , address=145.79.227.29:4942 , piece_id=UGG5ZUP62B6N7VB5O2TDLJ3CYVWVUQEHZOY362KXPV7NDGFOTYUA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471033848 (473h54m50.342116883s) [noise=true , address=145.79.227.33:4871 , piece_id=5WE37TDD26EFHRPFPTZG4KUEXQCOEFUKIGQJUXXT5TBFWAJHSQLQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471502841 (473h54m50.342585877s) [noise=true , address=145.79.227.39:4013 , piece_id=D3XGGDEJDYKRNPYM3SKASVQPEYC5LJ5UXSPJQMW5KII2RLYBP2QA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471852267 (473h54m50.342935302s) [noise=true , address=145.79.227.37:4565 , piece_id=CTCRPDXKNRPZLI35HE4O2N7JHJYGGSYXNPSTIVQ23XBXBF43VMEA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472267968 (473h54m50.343351004s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472596213 (473h54m50.343679248s) [noise=true , address=145.79.227.41:4086 , piece_id=CVHACN2KJAUKOOGR7MTCMYT725L4DN4LCIUWNGK5JBXJAMNB72WA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472902737 (473h54m50.343985772s) [noise=true , address=145.79.227.8:4330 , piece_id=ZPJLYRARNKIJMMJ5EKCJXCI3CROCTJFT6ERHZ2UZTXQMW6SYWBYA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.473202198 (473h54m50.344285233s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.473549970 (473h54m50.344633005s) [noise=true , address=145.79.227.50:4374 , piece_id=FGY3UU2VI342L2GMBJBKD3I4TJTUY65AQ2AJENRM2DSH2MTWHVGA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.475125379 (473h54m50.346208414s) [noise=true , address=145.79.227.43:4159 , piece_id=CONWY4VFJO44KLFRJJHFKYBJIENXR5OZPWCANWFR23CPHAO5XHMA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.476176541 (473h54m50.347259586s) [noise=true , address=145.79.227.15:4875 , piece_id=SFECL2JQUYDHMJJNBXMHOZML25G6HF4246UDPSOGXBJ4BXEIC4ZA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.477297886 (473h54m50.348380921s) [noise=true , address=45.91.116.9:4261 , piece_id=CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.478326435 (473h54m50.34940947s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.479334314 (473h54m50.350417349s) [noise=true , address=145.79.227.51:4410 , piece_id=XXQRFI64ML5YCVX6GIEILCMINWD77IK2ORGOKV67RMSRYYG5F3SA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.480341863 (473h54m50.351424898s) [noise=true , address=45.91.116.6:4974 , piece_id=VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.482939428 (473h54m50.354022463s) [noise=true , address=145.79.227.45:4229 , piece_id=CTVIS33TC3CWQADPBXH2QTE4YHHNPSDQSUEFROENDMVY7XO7C3TA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:52.139736479 (473h54m52.010819514s) [upload=415994]: Next returned nil segment Dec 25 09:23:52.139763281 (473h54m52.010846316s) [upload=415994]: Waiting for error group managing segments... Dec 25 09:23:52.476853834 (473h54m52.347936869s) [noise=true , address=145.79.227.37:4565 , piece_id=CTCRPDXKNRPZLI35HE4O2N7JHJYGGSYXNPSTIVQ23XBXBF43VMEA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.478814327 (473h54m52.349897372s) [noise=true , address=145.79.227.41:4086 , piece_id=CVHACN2KJAUKOOGR7MTCMYT725L4DN4LCIUWNGK5JBXJAMNB72WA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.483839445 (473h54m52.3549225s) [noise=true , address=145.79.227.50:4374 , piece_id=FGY3UU2VI342L2GMBJBKD3I4TJTUY65AQ2AJENRM2DSH2MTWHVGA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.487109452 (473h54m52.358192497s) [noise=true , address=145.79.227.8:4330 , piece_id=ZPJLYRARNKIJMMJ5EKCJXCI3CROCTJFT6ERHZ2UZTXQMW6SYWBYA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.491064052 (473h54m52.362147097s) [noise=true , address=145.79.227.15:4875 , piece_id=SFECL2JQUYDHMJJNBXMHOZML25G6HF4246UDPSOGXBJ4BXEIC4ZA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.494245760 (473h54m52.365328795s) [noise=true , address=145.79.227.45:4229 , piece_id=CTVIS33TC3CWQADPBXH2QTE4YHHNPSDQSUEFROENDMVY7XO7C3TA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.494657655 (473h54m52.36574069s) [noise=true , address=145.79.227.43:4159 , piece_id=CONWY4VFJO44KLFRJJHFKYBJIENXR5OZPWCANWFR23CPHAO5XHMA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.499742347 (473h54m52.370825382s) [noise=true , address=145.79.227.39:4013 , piece_id=D3XGGDEJDYKRNPYM3SKASVQPEYC5LJ5UXSPJQMW5KII2RLYBP2QA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.502818174 (473h54m52.373901249s) [noise=true , address=145.79.227.29:4942 , piece_id=UGG5ZUP62B6N7VB5O2TDLJ3CYVWVUQEHZOY362KXPV7NDGFOTYUA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.520302817 (473h54m52.391385852s) [noise=true , address=145.79.227.51:4410 , piece_id=XXQRFI64ML5YCVX6GIEILCMINWD77IK2ORGOKV67RMSRYYG5F3SA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.752925748 (473h54m52.624008783s) [noise=true , address=45.91.116.9:4261 , piece_id=CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.882738941 (473h54m52.753821976s) [noise=true , address=145.79.227.33:4871 , piece_id=5WE37TDD26EFHRPFPTZG4KUEXQCOEFUKIGQJUXXT5TBFWAJHSQLQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.990394505 (473h54m52.86147755s) [noise=true , address=45.91.116.6:4974 , piece_id=VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.990414122 (473h54m52.861497167s) [seg_pos={0 0}, upload=415994]: Segment reached optimal threshold of 13 pieces. Dec 25 09:23:52.990564689 (473h54m52.861647724s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection Dec 25 09:23:52.990569888 (473h54m52.861652924s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990691491 (473h54m52.861774526s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 25 09:23:52.990696901 (473h54m52.861779946s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990859900 (473h54m52.861942935s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 25 09:23:52.990865131 (473h54m52.861948166s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990880981 (473h54m52.861964016s) [seg_pos={0 0}, upload=415994]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:23:52.990954581 (473h54m52.862037626s) [upload=415994]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:23:52.990961794 (473h54m52.86204483s) [upload=415994]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:23:53.296758561 (473h54m53.167841606s) [upload=415994]: Done uploading segments... Dec 25 09:23:53.296784670 (473h54m53.167867705s) [upload=415994]: Upload finished. err: <nil> Duration: 5.90898915s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.511271653s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:27:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.511294305s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)